# PrivaTools — notes for agents Facts about this repo that are expensive to rediscover. Everything here was learned by getting it wrong first. ## Verification - **Type-check with `npx tsc --noEmit -p tsconfig.app.json`.** A bare `npx tsc --noEmit` resolves `tsconfig.json`, which has `files: []` and compiles *nothing* — it passes while real errors sit in the tree. The `-p` form is what CI runs. - **Run the whole backend suite, not the files you touched.** A route-coverage test asserts every backend POST is either a registered tool or a named account endpoint, so adding an endpoint fails a file you never opened. - **Isolate backend test modules on macOS.** Native-library interactions across modules have killed whole-suite runs; CI on Linux runs it together. Use the pinned Python 3.12 `.venv` locally. The earlier standalone crash in `test_phased_routes.py` is fixed (2026-09-13): QR/barcode decoding now runs in bounded subprocesses with an OpenCV fallback when host zbar is broken. That module must pass locally too; a native crash is no longer an accepted baseline. See `backend/tests/test_qr_reader_isolation.py`. - **Fake PDFs in tests need an object.** Every PDF route's pre-check (`validate_pdf_content`) refuses a `%PDF-` header with no `obj` in its first 256 KB, in the words the parsers use for a corrupt file, so a header-plus-filler fixture stops there without reaching the code under test. Start such fixtures with `%PDF-1.7\n1 0 obj\n`, unless the pre-check is what you test. - **CI does not run on a plain branch.** `test.yml` and `security.yml` trigger on pull requests and pushes to `main`. To verify a branch without a PR: `gh workflow run test.yml --ref `. - **Five checks are required on `main`:** "Backend tests (pytest)", "Frontend tests (vitest)" and "Image builds and serves (docker)" from `test.yml`, "Frontend audit and build" and "Python dependency audit" from `security.yml`. Two steps inside the vitest job fail it on their own: "Check generated content is current" (reruns `npm run gen:llms`, fails on any diff) and, on pull requests only, "Check tool review dates". The image job boots the built image with `scripts/ci/probe-image.py`, and `release.yml` gates on the whole of `test.yml`. - **OpenSSF Scorecard fails on a `workflow_dispatch` run from any branch but `main`** — "Only the default branch main is supported". On pull requests its job passes and the "Scorecard" code-scanning check shows as skipped. Neither is a code problem. ## Current UI and product context - Air and Play are the two supported experiences. Their implementation is in `frontend/src/skins/experience/`; maintain the design and product decisions in `frontend/DESIGN.md` and `frontend/PRODUCT.md`. - `frontend/src/skins/daylight/SkinApp.tsx` remains a handwritten shared base integration. It has `@ts-nocheck`, so verify changed bindings in the browser. - `frontend/src/styles/skins.css` remains generated by `frontend/scripts/skin-palettes.mjs` through `build-skins.mjs`; keep generator output reproducible. Historical skin/redesign plans are not current guidance. - Documentation starts at `docs/README.md`; maintenance commands are grouped under `scripts/` and documented in `scripts/README.md`. ## Counts come from the registry The tool total is `tools.length + nonPdfTools.length` (`TOTAL_TOOL_COUNT` in `frontend/src/data/site-stats.ts`). Never write it as a literal — the site once advertised 221 when it had 219. The guards are narrower than they look: `src/test/skins.test.ts` fails on a number before "tools" in rendered text or a string only in `skins/daylight/SkinApp.tsx`; the Air/Play shells in `skins/experience/` and the README are not scanned. The few literals that must exist — `public/manifest.json`, `public/opensearch.xml`, `public/samples/sample.json` and any "NNN tools" in blog copy — are checked against `TOTAL_TOOL_COUNT` by `tool-registry.test.ts` and `seo-static.test.ts`, so a new tool means editing them by hand. ## Accounts - The consumer app defaults to **Clerk**, with email verification and password recovery. Configure the same Clerk publishable key for frontend and backend; localhost needs a development key. Unconfigured sign-in must be shown as unavailable, never silently switched to native recovery-code signup. - Clerk password policy is 12–72 characters with breached-password blocking, no mandatory character classes. Username is optional (4–64 characters). Passkeys use device verification; never claim biometrics leave the device. - Native credential mutations return 409 when Clerk is configured. - Legacy self-hosted accounts are opt-in with `VITE_AUTH_PROVIDER=local`. **Those accounts have no email path**. Their signup/replacement recovery code is the only way back; keep the mandatory save acknowledgement intact. Native-auth tests select this mode explicitly in the Vitest configuration. - Password hashing is `hashlib.scrypt` via `auth/hashing_pool.py`, which offloads to a thread pool — never call it on the event loop. - Per-account lockout lives beside the per-IP limiter. Any endpoint that verifies a secret needs **both**. ## Deploy Production uses Docker Compose on the existing shared Oracle ARM VM (2 cores, 12 GB RAM), behind host nginx. PrivaTools binds `127.0.0.1:8000`. Keep maintenance confined to this application's containers, images and data. CI builds and signs release images. The Oracle deployment script defaults to release-tag mode after the first release exists; merging `main` alone does not ship a release. Verify the selected Git revision, image signature, `/readyz` and running build revision. Use `deploy/oracle-vm/` and the deployment guides for current commands; do not rely on an old version recorded in memory. Durable accounts and API keys live in `app-data`, separately from temporary processing files in `app-temp`. Backup tooling and its timer are installed. Preserve both durable data and a usable rollback image during cleanup. Check live timer and backup state before maintenance. **Deploys are zero-downtime (2026-09-18).** `deploy/oracle-vm/rollout.sh` (installed as `privatools-rollout`) starts the new release as a second compose project, `privatools-interim`, on `127.0.0.1:8001`. Before moving traffic there it requires three things: `/readyz` reports the build, `scripts/ci/probe-image.py --running` passes, and the new job supervisor holds the queue. It then lets the old container drain, recreates `privatools-privatools-1` on 8000 and switches back. The steady state never changes: the backup script, the CI probe and the runbooks all address that container and port. Replacing the container with a bare `docker compose up` brings back the old outage; it is only for rolling back to a release older than the drainable supervisor, and only from the steady state (upstream on 8000, no interim). A degraded cut-over has its own back-out in `deploy/README.md`: there, that command would start v2.6.1 while nginx kept routing to the interim. A failed release never takes traffic, so there is nothing to roll back; `privatools-rollout --rollback` is itself zero-downtime, also from the degraded state. Details, evidence and the cut-over runbook are in `deploy/README.md`. - **Run deploy scripts as ubuntu, never root.** `rollout.sh`, `auto-deploy.sh` and `deploy.sh` refuse root. The deploy lock lives in sticky `/tmp`, and `fs.protected_regular=2` makes root's open of ubuntu's file fail; a root run before the file exists would create one the timer can't open. By hand: `sudo runuser -u ubuntu -g ubuntu -G docker -- privatools-rollout ...`. - **Rollout exit codes drive the timer.** 1: the release is at fault, marked failed until a newer tag. 2: a host problem or unmet precondition. 3: degraded, the interim serves. 4: nginx or a still-routed container needs a human. The timer retries 2, 3 and 4 only after a 10-minute backoff. Classify new failure paths accordingly; a host problem reported as 1 blocks the release for good. - **nginx is verified, not trusted, and otherwise left alone.** Each switch is recorded (`.privatools-deploy.switching`) before the helper runs and cleared once nginx visibly retires its old worker generation. A run that finds one recorded re-applies the file's port, because a kill between the helper's rename and reload leaves them apart; no other run reloads the shared nginx. Traffic never falls back to an interim the file does not name: it may not have passed the gates. A container that still receives requests after the generation this deploy retired exits is never removed (exit 4). - **One failed Docker call proves nothing.** Status polls and `docker inspect` are asked again until their deadline; a supervisor or container is judged failed only from an answer. In the cut-over deploy, a container nginx routes to is stopped only if it actually restarted; otherwise the run ends degraded. Docker 29's CLI words a missing object in lower case (`error: no such object`), so match `[Nn]o such`, never `No such` alone. - **Old and new code share the SQLite database for the overlap** (about a minute), and the new container applies its migrations on start. Migrations must be additive and quick (one `BEGIN IMMEDIATE` against a database the live release writes), with the previous release still working on the new schema. Never add a column to a table written with a positional `INSERT ... VALUES` (`api_async_worker`, `api_async_ingest`, `api_async_submit_window`, `api_v1_leases`, `api_v1_rate_buckets`): the running release's inserts would fail. A failed `store.init()` is only logged ("accounts disabled"); neither readiness nor the probe checks the store. - **Exactly one job supervisor holds `jobs/worker.lock`.** A second one waits as a standby. It must never exit, because the launcher stops the whole container when a child exits. SIGUSR1 to the container (the launcher relays it) drains: the supervisor finishes its job, then releases the lock. SIGUSR2 resumes. The launcher starts children with both ignored until the worker installs its handlers. Readiness counts this container's live standby through `backend/app/job_handover.py`'s state file in its private `/tmp`, because the shared heartbeat names the old build until the handover: without limit while another supervisor serves the queue, for at most `queue_seconds` otherwise. The deploy polls `python -m backend.app.job_handover --status`, which is standard library only on purpose: keep FastAPI and pydantic out of it. - **The only root step is the nginx switch.** It runs through `sudo -n /usr/local/sbin/privatools-nginx-upstream set 8000|8001`. The helper refuses a port where nothing is ready, serializes concurrent switches, runs `nginx -t` and restores the old upstream on failure. It also ignores SIGTERM from rename to reload. The deploy unit must not set `NoNewPrivileges`. - **The installed rollout reads a release's files** (`deploy/README.md` lists them): the probe's `--running` command line and `PRIVATOOLS_PROBE_HOST`, `compose.interim.yml` and its volume variables, `PRIVATOOLS_HOST_PORT`, the status JSON and the signal meanings. Change them only compatibly. - **Installed scripts are copies.** A deploy resets the checkout, not `/usr/local/bin`. Reinstall with `install-auto-deploy.sh`, which never starts a deploy unless given `--start`; the backup script is installed separately and its timer is never touched. - **Image retention (2026-10-05).** After exit 0 only, `--rollback` included, the rollout removes PrivaTools' own release images by name, never forced or pruned. It keeps the newest `KEEP_IMAGES` (3), every container's image and the rollback record, and a failure there never changes the exit status. The API remains free with bounded fair usage on this server. Async job results expire within one hour and can be explicitly deleted immediately. ## Read-only container `read_only: true` is on. Only LibreOffice needs to write outside the volumes: it puts its IPC pipe in `/tmp` and fails with "no valid pipe path found" otherwise, and it **ignores `TMPDIR`** for that (TMPDIR is already `/app/temp`), so the tmpfs is the only fix. weasyprint, rembg and the image tools all pass with no writable `/tmp` at all. The baked caches live in `/app/cache`, not `/tmp`, because the tmpfs masks whatever is under it — when they were on `/tmp`, rembg silently re-downloaded u2netp from a GitHub release on the first request and the failure looked like latency. If you move them back, that returns. onnxruntime reports an unreadable model as a bare `system error number 13`, naming neither the file nor the permission, so the `chown` of `/app/cache` is load-bearing. ## Analytics (changed 2026-09-24) - Google Analytics is **default-on for every visitor**; the Privacy page switch is the only opt-out. No consent prompt, no regional policy, and DNT or GPC are not read. `deploy/analytics.md` is the runbook. - **Automated browsers are never measured:** `navigator.webdriver` true, or a `HeadlessChrome`/`PhantomJS` user agent, keeps the tag unloaded. No screen-size or country heuristics. A Playwright check of analytics must therefore launch Chromium with `--disable-blink-features=AutomationControlled` and an ordinary user agent, or it sees no tag at all. - **Only the first page view of a page load says where the visit came from:** the external referrer's origin and the five `utm_` tags, sanitized, read when the beacon module loads. Never put them in `gtag("set")`: the global defaults feed every automatic event and later hit. - **Every tool surface that processes files must call `emitToolRun`** from `lib/toolRun.ts` at its success and failure points, unless it runs through `GenericUI`, `SimpleConvertUI`, `useMultiFileProcessor` or `useMediaJob`, which report centrally. `src/test/tool-run-coverage.test.ts` walks `components/tool-ui` and fails on any file that talks to the backend without it. The beacon derives the slug from the route and drops unknown slugs. - **A failure report passes what it caught:** `emitToolRun({ outcome: "error", ... }, e)`, or the run's first failure for multi-file engines, or an explicit `errorKind` when the tool knows better. `toolErrorKind()` classifies from `__status`, the `__kind` tag `lib/api.ts` puts on status-less failures (`withErrorKind` for a tool's own throws), error names and BYOK error kinds; only the fixed category reaches the beacon. Parse a server's JSON with `readJson` from `lib/api.ts`, not `res.json()`, so a body that does not parse counts as `server` rather than `browser`. A user cancel is not a failure: cancelled files are not counted, and an error caused by a cancel is dropped. The coverage test also fails on a failure report with neither. `toolRun.ts` matches BYOK errors by name and must never import `lib/byok`, or every tool page would need the provider CSP. - React Router navigations reach the beacon through `notifyNavigation()` in the app root; pushState fires no popstate, so page views were once lost. ## AI stack (added 2026-09-01) - **Two AI paths, one dialog.** On-device models (transformers.js → browser Cache API "transformers-cache") and BYOK (`lib/byok/*`). The top-bar AI hub (`AiHubDialog`) manages both; `lib/localModels.ts` is the model registry and does honest cache introspection — never invent an "installed" state. - **BYOK client is three functions** (`complete`, `transcribe`, `countTokens`) in `lib/byok/client.ts` — the only network calls in the package, on purpose. Message content accepts image parts; `buildRequest` maps them per provider shape. Tasks (`tasks.ts`) fence document text per call — keep that. - **CSP is per-path and guarded by tests.** `_BYOK_PATHS` must equal the pages whose components can transitively import `lib/byok` — `test_byok_csp.py` WALKS both ToolPage and NonPdfToolPage to verify. `_WASM_EVAL_PATHS` covers every page running onnx/tesseract wasm; `_TESSERACT_PATHS` additionally extends script-src to cdn.jsdelivr.net (blob workers inherit page CSP). A new AI page that misses these ships broken ONLY in prod — dev has no CSP. - **Non-PDF tools live at `/tools/`** (NonPdfToolPage), PDF at `/tool/`. The Daylight skin mounts the right switch via `withRealTools` — both, since 2026-09-01; it once only knew ToolPage and every non-PDF tool rendered wrong in the skin. - **`useMultiFileProcessor` reads state through a ref mirror** (`mutate()`). Never read state by capturing values inside a `setState` updater — React defers updaters and the hook silently processed zero files for months. ## Registering a tool A new slug touches about a dozen places. The count, registry, route-coverage, CSP-walker and guide-export tests catch most misses. - **Registry entry** in `frontend/src/data/tools.ts` (PDF, `/tool/`) or `non-pdf-tools.ts` (`/tools/`), with a `seoTitle` (40–60 chars, query-first, no brand, unique), a `metaDescription` (120–160 chars, ends with a period, unique), `synonyms` and `lastReviewed`, all enforced by `frontend/src/test/tool-registry.test.ts`. Search reads the entry's `synonyms`; the old `SYNONYMS` map in `components/CommandPalette.tsx` is not mounted anywhere. - **UI:** `ToolUI` in `pages/ToolPage.tsx` / `NonPdfToolPage.tsx` falls back to `GenericUI`, so add a case only for a dedicated component. `ToolIllustration` is imported there but rendered nowhere; it needs no entry. A backend path other than `/` goes in `frontend/src/lib/tool-endpoints.ts`. - **Backend:** the route's router is included in `backend/app/main.py` under `/api`, and in the `api_v1.mount` list to appear under `/api/v1`. `test_route_coverage.py` fails when a server-backed tool has no route, when a declared route is not included in the app, and on a POST that is neither a registered tool nor a named helper or account endpoint. - **Damaged and locked PDFs:** a route that takes an upload goes in `ROUTES` (a PDF) or `NOT_PDF` of `backend/tests/test_damaged_pdfs_everywhere.py`, which posts cut-short and locked PDFs to every route in `ROUTES` and fails on any 5xx. Open the upload with `open_pdf_document` or `process_pdf` (`utils/cleanup.py`), not a bare `fitz.open`, which opens a locked PDF and fails later, and with pikepdf through `open_pikepdf` or `safe_open_pdf`: these refuse a repaired PDF that lost pages, which the libraries otherwise read as a shorter file (`utils/declared_pages.py`, `test_pages_lost.py`). Let `ToolError` through, and have the catch-all ask `pdf_read_error` (`utils/pdf_errors.py`) before it answers 500. - **PyMuPDF's switches can be process-wide.** `small_glyph_heights`, which `find_tables` turns on, is a C global that every request thread's text extraction reads: it changes only through `find_tables` and `exact_glyph_boxes()` in `backend/app/utils/tables.py`, which share one lock, and code whose glyph boxes must be exact holds `exact_glyph_boxes()`, as Smart Redact does (#340). That lock covers no other `TOOLS.set_*` switch, so the web process changes none. - **Guide:** steps and FAQ in `backend/app/tool_content.py`, then `.venv/bin/python scripts/seo/export-tool-guides.py` regenerates `frontend/src/data/tool-guide/*.json` (Python is authoritative, `test_tool_guide_export`). - **CSP sets** for a page that runs a model or a BYOK provider (AI stack above). - **Generated files:** run `npm run gen:llms` in `frontend/` and commit what it writes (the llms, sitemap, feed and content JSON in `public/`, plus `src/data/tool-blog-links.json`). The vitest job's "Check generated content is current" step reruns it and fails on any diff (#178). - **`seo_meta.py` holds no tool copy (#179).** Its `_PDF_TOOLS`/`_NONPDF_TOOLS` are built at import from the manifest gen-llms writes: the build's `tool-content.json`, or the committed `frontend/public/tool-content.json` without a build. With neither readable the app refuses to start. Never hand-copy registry text into Python (a hand copy drifted on most tools). So a worktree without a build runs the tool-manifest tests too; the blog-link test still needs `npm run build`. `test_tool_registry_parity.py` compares the manifest with the registry source, but in CI the backend job's build regenerates the manifest before pytest, so only the frontend job's step above catches a stale commit. The sitemap reads the manifest as well, so nothing in `sitemap.py` is edited per slug. - **Count literals:** the hand-kept totals in "Counts come from the registry". - **`lastReviewed`** moves only when that tool's own copy changes; never bulk-bump, because a sitemap where every date moves together is discounted. `frontend/scripts/check-review-dates.mjs`, a `pull_request`-only step in `test.yml`, fails when a tool's `seoTitle`, `metaDescription`, `longDescription` or `description` changed without its date moving, unless that date already falls on or after the day before the change began (the author date of the branch's oldest commit touching a registry, or today for uncommitted edits), or when more than 25 dates move without `[bulk-review]` on its own line (the first text on a line) of the PR title or a commit message; a mid-sentence mention does not count. Its limit: a long-lived branch can pass with a date a few days old. - **Sitemap priority:** `frontend/src/data/sitemap-priority.json` lists the tools that get priority 0.8 — head PDF tools plus developer tools chosen because their search results are winnable niches; keep it short and reviewed, not a wish list. ## Dependencies (changed 2026-09-18) - **Python pins live in `requirements*.in`; `requirements*.txt` are hashed locks compiled from them (#208).** The image installs `requirements.txt`; CI and `npm run setup:backend` install `requirements-dev.txt`, which starts with `-r requirements.in`; CI's own tools come from `requirements-ci.txt`. All install with `--require-hashes`, so editing a `.in` alone ships nothing. To add or bump a package, edit its pin in the `.in` and rerun the `uv pip compile --generate-hashes --universal --python-version 3.12 …` command from the header of every lock that includes it (a runtime pin is in both `requirements.txt` and `requirements-dev.txt`), with `-P ==` so uv moves nothing else. `test_requirements_lock_sync.py` fails when a pin and its lock disagree, a lock lacks hashes or its exact header, or Dependabot is not on `uv`; `test_declared_dependencies.py` fails when code imports a package that `requirements.in` does not declare. - **Check arm64 before bumping a native package.** The release image is arm64 and PR CI builds only amd64. `pip download --only-binary=:all: --platform manylinux_2_28_aarch64 --python-version 3.12 --no-deps ==` must find a wheel; ask for manylinux_2_28, because pikepdf, onnxruntime and pillow-heif ship no manylinux2014 wheels and that platform finds nothing. - **Dependabot updates Python through its `uv` ecosystem,** which edits the `.in` pin and reruns each lock's header command. The `pip` ecosystem never regenerated the locks, so every one of its PRs failed the lock test. Minor and patch updates arrive as one grouped PR, because each regenerates the same locks. - **npm minor and patch updates arrive as one grouped PR per run too;** majors come one per PR. Every change to `.github/dependabot.yml` on `main` starts an immediate run of every ecosystem, besides the weekly schedule: on 2026-09-18 six such changes set off five waves of single-package npm PRs, 24 in all, each wave refilling whatever of the five open-PR slots had been freed. - **Held majors.** `.github/dependabot.yml` ignores these until a dedicated migration, with the reason beside each rule; delete the rule when that work starts. - **React 19:** `react`, `react-dom`, `@types/react` and `@types/react-dom` must move together, with every React-dependent library checked in a browser; `react-dom` 19 alone cannot even install (#225). - **eslint-plugin-react-hooks 7:** from 6 on, its recommended set carries the React Compiler rules: 77 errors in 54 files here, `useMultiFileProcessor` among them. Switching the rules off is not the fix (#225). - **ESLint 10** (`eslint`, `@eslint/js`): eslint-plugin-react-hooks declares ESLint 10 support only from 7, so both wait for that task (#247). - **Tailwind CSS 4 with tailwind-merge 3:** the PostCSS plugin moved, configuration moves into CSS and utilities the UI uses are renamed, so it needs visual checks of Air and Play; tailwind-merge 3 supports only Tailwind 4 (#231). - **pdf.js 6** (`pdfjs-dist`): it removes APIs the PDF tools call, such as `PDFDocumentProxy.destroy`, and every tool that renders or reads PDFs in the browser needs a browser check (#236). 5.x minors are held too: 5.6.83 up to 6.2.108 carry GHSA-hq66-cqwq-w95j, so stay on 5.5.207 and go straight to 6.2.108 or later. - **TypeScript 6 and 7** (`typescript`): typescript-eslint 8.70 supports only TypeScript below 6.1, and 7 is the native Go compiler, so the compiler, typescript-eslint and the CI type-check move together in one task (#260). - **The `python` base image** ignores minor and major updates: moving Python is a migration, because the locks are compiled for 3.12. Without the rule Dependabot targeted only the newest tag and proposed no digest refresh at all (#203). - **Don't `@dependabot rebase` a Docker digest PR within three days of the tag's last push.** Dependabot's default 3-day cooldown then reports the image as up to date and closes the PR (#152); merge it as it is, or wait (#203). - **Releases are signed with cosign v2.6.5** (`cosign-release` in `release.yml`). cosign-installer v4 defaults to cosign v3, which signs in the bundle format; a verifier older than v2.6.3 on the server finds no signature and the deploy refuses the release. Drop the pin once `cosign version` on the server reports v3 (v3.1.3 or later also fixes GHSA-fx35-mq7g-6g98) (#184). - **The frontend audit allows listed exceptions only** (`frontend/audit-exceptions.json`, run by `scripts/audit-check.mjs` in place of `npm audit --audit-level=high`). An exception names one advisory, the packages it may cover, why the vulnerable code can't be reached, and an expiry date. It stops covering its advisory on that date or once npm reports a non-major fix; any other high or critical advisory still fails the build. Use it only when the only fix is a held major, and never more than 90 days out (the check refuses a malformed or longer expiry, and a report it can't read). The first one (GHSA-vfj7-8cjw-p6xm: braces via tailwindcss 3, also as tailwindcss-animate's peer, so npm counts it as production, though it runs only at build time) expires 2026-11-15; Tailwind 4 removes it. - **npm lockfile rule:** regenerate `package-lock.json` with the npm CI runs, not a local one. CI pins Node 26 and uses its bundled npm, which a job log shows under "Environment details" (11.19.1 on 2026-09-18), e.g. `npx -y npm@11.19.1 install --package-lock-only`. Local npm 11.6 prunes `@emnapi/*` platform entries and breaks CI's `npm ci`.