# MeisterTracker Deployment Guide

This guide walks through setting up MeisterTracker on a fresh Debian server. It has been verified end-to-end on Debian Trixie.

> **TL;DR:** SSH-clone the repo, install Node 22 + PM2, generate a self-signed cert, start. Optionally configure DuckDNS + Let's Encrypt via the admin UI for a real cert and public access.

## 1. System Requirements & Dependencies

Run the following commands as `root` or with `sudo`.

### Update System
```bash
sudo apt update && sudo apt upgrade -y
```

### Install Core Dependencies
MeisterTracker requires **Node.js 22+** and uses SQLite via the built-in `node:sqlite` module (no external SQLite installation needed).

```bash
sudo apt install -y curl ca-certificates git openssl
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt install -y nodejs
```

Verify the Node.js version:
```bash
node --version   # must be v22.0.0 or higher
```

> **Nginx is NOT installed by default.** Only install it if you choose Path B (Section 7) — the app has its own HTTPS and Let's Encrypt integration that does not need a reverse proxy.

### Install Process Manager
PM2 keeps the server running and auto-restarts on crashes. Global npm installs require `sudo` — without it you'll get `EACCES` errors trying to write to `/usr/lib/node_modules`.
```bash
sudo npm install -g pm2
```

## 2. Project Setup

### Option A — Clone via SSH (recommended)

If you already have an SSH key registered with GitHub, skip the keygen. Otherwise:

```bash
ssh-keygen -t ed25519 -C "your-email@example.com"
cat ~/.ssh/id_ed25519.pub
# Copy the printed key, then add it at https://github.com/settings/keys
ssh -T git@github.com   # should greet you by username
```

Prepare the directory and clone:
```bash
sudo mkdir -p /var/www
sudo chown $USER:$USER /var/www
git clone git@github.com:meisterpilze/meistertracker.git /var/www/meistertracker
cd /var/www/meistertracker
```

### Option B — Clone via HTTPS

GitHub no longer accepts password authentication for git operations. You'll need a **Personal Access Token** (https://github.com/settings/tokens) — paste the token when git prompts for a password.

```bash
sudo mkdir -p /var/www
sudo chown $USER:$USER /var/www
git clone https://github.com/meisterpilze/meistertracker.git /var/www/meistertracker
cd /var/www/meistertracker
```

### Install Dependencies
```bash
npm install --production
```

## 3. Environment Configuration (Optional)

Create a `.env` file in the project root (`nano .env`) — every variable has a sensible default, so an empty file is fine:

```ini
# Server port (default: 3000)
PORT=3000

# Log format: "json" (default) or "text"
LOG_FORMAT=json

# Port for HTTP -> HTTPS redirect. Default: 80.
# Binding port 80 needs root; if it fails the server logs a warning and
# stays in HTTPS-only mode. The same-port redirect on PORT keeps working.
HTTP_REDIRECT_PORT=80

# Set to true ONLY when behind a reverse proxy (e.g. Nginx, Section 7).
# DO NOT set to true without a proxy — it allows clients to spoof their
# IP via X-Forwarded-For headers.
# TRUST_PROXY=false

# Windows-only: printer name for label printing (no effect on Linux)
# PRINTER_NAME=ZDesigner GK420d

# Windows print bridge — see Section 10. When set, the server forwards
# label prints + status checks to scripts/print-bridge.ps1 running on a
# Windows PC. Leave unset to use the ZPL-download fallback instead.
# PRINT_BRIDGE_URL=https://<windows-pc-ip>:9100
# PRINT_BRIDGE_TOKEN=<long-random-string>

# Outbound harvest feed — see Section 15. Unset means off.
# HARVEST_WEBHOOK_URL=https://example.org/harvest
# HARVEST_WEBHOOK_SECRET=<long-random-string>

# Photos. Experiment photos keep their full-resolution original (a few MB
# each) next to the 1280 px picture the app shows. The folder cap is the
# outer bound (default 100 GB); what protects the disk is the free space
# that has to remain after every original (default 2 GB). The pictures the
# app shows, contamination photos among them, still go through below that,
# down to 256 MB free, so a report is not refused for want of room for
# originals. A refused photo is answered 507 and the app says the storage
# is full. Below twice the reserve the log warns once an hour ("Photo disk
# getting full"), and /api/health/full shows photosReserveMB and photosLow.
# PHOTO_DIR_MAX_GB=100
# PHOTO_MIN_FREE_GB=2

# Lab notebook: the daily RFC 3161 timestamp over the history's hash chain
# (Section 9). Only a SHA-256 value leaves the server. Default: a public
# timestamp authority over HTTPS; "off" turns stamping off. A worktree
# instance does not stamp unless this is set. The app checks every stamp's
# signature up to a root certificate Node ships; another authority whose root
# is not among them gets its stamps reported as unchecked (Section 9).
# LAB_TSA_URL=https://timestamp.sectigo.com
```

## 4. TLS Certificate Setup

HTTPS is required for camera-based QR scanning (iOS Safari enforces this). Generate a self-signed certificate:

```bash
bash gen-cert.sh
```

This creates `certs/server.key` and `certs/server.crt`, valid for 365 days, covering `localhost`, your LAN IP, and `127.0.0.1`. That's enough for LAN access.

To include a custom domain in the cert, pass it as an argument — replace the placeholder with your real domain:
```bash
bash gen-cert.sh <your-domain>
```

> **Note:** If you'll use Path A below (DuckDNS + Let's Encrypt), skip the domain argument here. The app will obtain a real Let's Encrypt cert that overrides this self-signed one.

## 5. Start the Server

The recommended way to manage the server is the `update_server.sh` script:

```bash
# First start
bash update_server.sh start

# Update from GitHub, back up data, and restart
bash update_server.sh

# Stop the server
bash update_server.sh stop

# Check status
bash update_server.sh status

# Regenerate TLS certificate
bash update_server.sh gen-cert
```

This starts the server as a PM2 process named `meisterpilze`.

### What an update does before it touches anything

`bash update_server.sh` does not simply swap the code and hope. Between
installing dependencies and restarting, it runs `scripts/preflight.js`, which
tries the new version against **a copy of your real database**:

- it snapshots the live file with `VACUUM INTO`, which does not write to the
  source, so this is safe while the server is up and serving;
- it runs any pending migrations **on the copy**, so a migration that throws
  against your six months of rows fails there instead of on the real file;
- it compares row counts table by table and refuses if anything shrank;
- it parses and loads every module the server loads, which is where a bad
  deploy usually dies;
- it arms and disarms the DuckDNS updater against the copy, without a network.

If any of that fails, **nothing is changed**: the old server is still running,
still serving, and the script tells you what broke and how to get the working
tree back. You can also run it yourself at any time:

```bash
node scripts/preflight.js
```

The GitHub webhook auto-deploy runs the same preflight, against the snapshot
of the database it takes first. See Section 9.

### And if the new version still will not start

The script waits for the restarted process and checks it stayed up. If it did
not, it now rolls back to the `stable` tag — the last commit that was known to
start — reinstalls dependencies and brings that version up again, rather than
leaving the server down until somebody reads the log. A rollback across a
migration is safe: an older build ignores columns it does not know about. One
trace stays: history entries an older build writes have no link in the lab
history's hash chain, and the next newer start links them and records that
(Section 9).

The `stable` tag is written by this script after every update whose restarted
server stayed up, so it only exists once one update has succeeded.
`bash update_server.sh start` never writes it.

### Enable PM2 Startup on Boot

`pm2 startup` only **prints** the command — you have to run it as root yourself:

```bash
pm2 startup
# Copy the printed `sudo env PATH=...` command and run it. Example output
# (yours will differ — use what your machine prints):
#   sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u <your-user> --hp /home/<your-user>
pm2 save
```

### PM2 Log Rotation (REQUIRED)

PM2 writes stdout/stderr to `~/.pm2/logs/meisterpilze-out.log` and
`meisterpilze-error.log`. Without rotation those files grow forever and
eventually fill the disk. Install the official rotation module once during
setup:

```bash
pm2 install pm2-logrotate
pm2 set pm2-logrotate:max_size 50M
pm2 set pm2-logrotate:retain 14
pm2 set pm2-logrotate:compress true
pm2 set pm2-logrotate:dateFormat YYYY-MM-DD_HH-mm-ss
```

What each setting does:

- `max_size 50M` — rotate the active log when it crosses 50 MB.
- `retain 14` — keep 14 rotated archives, then delete the oldest. With our
  log volume that's roughly 2–4 weeks of history; bump higher if you need
  longer audit retention.
- `compress true` — gzip rotated archives so 50 MB shrinks to a few MB on
  disk.
- `dateFormat YYYY-MM-DD_HH-mm-ss` — readable filenames sorted naturally.

These settings persist across reboots once `pm2 save` has been run.

### Verify
```bash
pm2 logs meisterpilze    # view server logs (Ctrl+C to exit)
curl -k https://localhost:3000/api/health
```

The first health check returns `{"error":"setup_required"}` because no admin exists yet. Open the app in a browser and the login screen will switch to setup mode automatically:

```
https://<server-ip>:3000
```

Accept the self-signed cert warning, create the first admin, log in.

---

# Public Access — Choose ONE

To reach the server from outside your LAN with a real (browser-trusted) TLS cert, pick **one** of the two paths below. They don't combine — Path B replaces Path A.

## 6. Path A — Built-in DuckDNS + Let's Encrypt (recommended)

The app has built-in DuckDNS dynamic DNS and Let's Encrypt certificate management. No extra software is needed on the server.

**What you get:**
- Public hostname like `<your-name>.duckdns.org`
- Real Let's Encrypt cert, auto-renewed
- DNS-01 challenge — port 80 is **not** required to be reachable
- URL: `https://<your-name>.duckdns.org:3000` (port stays in URL)

### Setup steps

1. **Register a DuckDNS subdomain** at https://www.duckdns.org/ and copy your token.
2. **Forward port 3000** from your router/firewall to the server.
3. **In the MeisterTracker admin UI** (Settings → DuckDNS):
   - Subdomain prefix only (without `.duckdns.org`)
   - Token from DuckDNS
   - Enable
   - Wait a few minutes for DNS propagation
4. **Enable Let's Encrypt** in the same admin section. The server obtains the cert via DNS-01 (using DuckDNS TXT records) and renews automatically.

That's it — no Nginx, no Certbot, no system-level services beyond what you've already set up.

### The gap the server cannot cover itself (recommended)

The server keeps the DuckDNS record current while it runs, retries when an
update fails, and checks against the authoritative nameservers that the record
it wrote is the record being served.

None of that happens while the server is **down**. A failed deploy, a reboot
where PM2 never came back, a crash nobody saw — the address changes anyway, and
the name goes on pointing at wherever the line used to be. By the time somebody
starts the server again, the address it advertises belongs to somebody else. The
symptom is a server that is unreachable for a reason unrelated to whether it is
running.

`scripts/duckdns-fallback.js` closes that. It runs from the system scheduler
rather than from the server, so it is still there when the server is not.

**It does not double the updates.** The server records the time of each of
*its own* successful updates in one column; the fallback reads that column,
never writes it, and records itself in another. It stands down while the
server's column is fresh, without opening a socket, and acts only once that
column is older than fifteen minutes — long enough that a server merely working
through its retry backoff is left alone rather than raced.

Keeping the two columns apart is what makes the admin banner still trustworthy:
a server whose own updater has died goes red even while the fallback is holding
the record up, and the banner says both.

**Linux (systemd):** start the server once first — the unit runs as whoever
owns `meistertracker.db`, and the installer refuses rather than guessing.

```bash
sudo bash scripts/install-duckdns-fallback.sh
```

That installs and starts `meistertracker-duckdns.timer` — every five minutes,
and two minutes after boot. Check it took:

```bash
systemctl list-timers meistertracker-duckdns.timer
```

Prove it works without waiting for a real outage (`--force` skips the
stand-down check):

```bash
sudo -u <the-user-that-owns-the-db> node scripts/duckdns-fallback.js --force
```

Read what it has been doing:

```bash
journalctl -u meistertracker-duckdns.service --since today
```

Remove it again with `sudo bash scripts/install-duckdns-fallback.sh --uninstall`.

`update_server.sh` checks after each deploy whether the timer is there, and
prints the command if it is not — but only on a machine where DuckDNS is
actually configured and systemd is present. It never installs anything itself:
it runs as the application user, and writing to `/etc/systemd/system` needs
root. This is the same shape as `pm2 startup` above, for the same reason.

**Windows (Task Scheduler):** double-click `install-duckdns-fallback.bat` and
answer the UAC prompt. That is the whole installation — the script asks Windows
for administrator rights itself, so it does not matter which shell you start it
from.

Administrator is needed to *register* the task and for nothing else: a task
registered by an ordinary user only runs while that user is signed in, which
misses the case this exists for — the machine rebooted and nobody logged in.
Running without a session needs S4U, and registering S4U needs elevation. The
task is registered `-RunLevel Limited`, so the recurring job itself runs
unprivileged; it reads a database file and makes one HTTPS request.

This is separate from `install-autostart.ps1` and does not replace it: that one
starts the server, this one covers the times it is not started. Check with
`Get-ScheduledTaskInfo -TaskName MeisterTrackerDuckDNS` — last result `0` is
good. Remove it with `install-duckdns-fallback.bat -Uninstall`.

> **Both installers refuse to run from a git worktree,** and so does the script
> itself. A worktree usually carries a copy of the same token, and a timer
> inside one would quietly fight the real instance over the same record.

**How you tell it is working:** Settings → DuckDNS. The banner goes red when
this server's own updater has stopped succeeding, when it is not running, or
when something else keeps overwriting the record — and it says which. When the
fallback is carrying the record it says that too, *underneath* the red: the name
still resolves, and the in-process updater still needs looking at.

Green means nothing is currently wrong that this server can see. It is not a
positive confirmation from the nameservers on its own — that check runs after a
change and on a slow rota, and the banner adds a line saying so when it has
actually run.

## 7. Path B — Nginx Reverse Proxy + Certbot (alternative)

Use this path **only if** you need:
- Clean URLs without `:3000` (port 443)
- Multiple services on the same server
- Advanced HTTP features (rate limiting, caching, header manipulation)

> **Important:** With Nginx in front, **disable** the app's built-in DuckDNS/Let's Encrypt (Path A) — they conflict on cert handling.

Install Nginx and Certbot:
```bash
sudo apt install -y nginx certbot python3-certbot-nginx
```

Create a configuration file at `/etc/nginx/sites-available/meistertracker` — replace `<your-domain>` with your actual domain:

```nginx
server {
    listen 80;
    server_name <your-domain>;

    # The largest request is an experiment photo's original, up to 40 MB.
    # Without this, nginx refuses every body over 1 MB with 413, and no
    # photo ever gets its original.
    client_max_body_size 50m;

    # Deny access to hidden files (except .well-known for SSL challenges)
    location ~ /\.(?!well-known) {
        deny all;
        access_log off;
        log_not_found off;
    }

    # Security headers
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "SAMEORIGIN" always;

    # Health check endpoint
    location /api/health {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Server-Sent Events (real-time sync)
    location /api/events {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 86400s;
        chunked_transfer_encoding off;
    }

    # All other requests proxied to Node.js
    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
```

Enable site, fetch SSL cert, restart:
```bash
sudo ln -s /etc/nginx/sites-available/meistertracker /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx
sudo certbot --nginx -d <your-domain>
```

Set `TRUST_PROXY=true` in your `.env` so the server reads client IPs from `X-Forwarded-For` headers, then restart:
```bash
bash update_server.sh
```

---

## 8. Security Hardening (Recommended)

### Prevent Brute Force Attacks (Fail2Ban)
```bash
sudo apt install -y fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban
```
Works automatically out of the box for SSH.

### Use SSH Keys
On your **local** machine (laptop/desktop, not the server):
```bash
ssh-keygen -t ed25519
ssh-copy-id -p <ssh-port> <user>@<server-ip>
```

Verify key-based login works (`ssh -p <ssh-port> <user>@<server-ip>` should not prompt for a password). Then optionally disable password auth on the server:
```bash
sudo nano /etc/ssh/sshd_config
# Set: PasswordAuthentication no
sudo systemctl restart ssh
```

> ⚠️ Verify key-based login works **before** disabling password auth, or you'll lock yourself out.

### Firewall
If your server isn't already firewalled at the network/hypervisor level (e.g. Proxmox host iptables), install UFW:
```bash
sudo apt install -y ufw
sudo ufw allow ssh
sudo ufw allow 3000/tcp     # Path A: app's HTTPS
# OR for Path B:
# sudo ufw allow "Nginx Full"
sudo ufw enable
```

## 9. Backups

MeisterTracker automatically creates daily SQLite backups at midnight:
- Stored in the `backups/` directory
- Uses `VACUUM INTO` for WAL-consistent snapshots
- Keeps the last 30 days
- Filename pattern: `meisterpilze_backup_YYYY-MM-DD.db`

### Snapshots before an update

Every way of updating takes a snapshot of the database before the new code starts, and with it any new migrations: `update_server.sh`, `START.bat`, and the GitHub webhook auto-deploy (`/api/webhook/github`). They go into `backups/` beside the daily files, with the pattern `meistertracker_YYYYMMDD_HHMMSS.db`. Both prefixes are recognised by the health check below. `START.bat` and `update_server.sh` take it with `scripts/backup-db.js`, which reads through the WAL. If that fails, they use the `sqlite3` CLI where one is installed, and as a last resort a plain copy of `meistertracker.db`. The copy misses whatever is still in `meistertracker.db-wal`, and the output says so in a WARNING. If not even the copy can be written, `update_server.sh` stops before the new version starts. The webhook deploy takes it inside the running server, through the server's own connection to the database.

The server keeps the newest 10 of these snapshots and deletes the older ones. It rotates them at every start, with the nightly backup run, and right after the webhook's own snapshot. The number is `PRE_UPDATE_SNAPSHOT_KEEP` in `scripts/rotate-backups.js`. "Newest" is the order the server first saw them in, kept in `backups/.pre-update-snapshots.json`, not the time in the file name: every writer names its snapshot after the PC's clock, and a clock that ran ahead, or one that runs behind after a dead CMOS battery, would otherwise make the snapshot just written look like the oldest. The newest snapshot is never deleted, and neither is any file less than a minute old. Snapshots that were already there the first time the rotation ran are kept for good, as this section used to promise; delete them by hand once you no longer need them. A ledger that cannot be read starts over the same way, keeping everything that is there. The daily auto-backups keep their own 30 days, and `.enc` files, photo archives and anything you have renamed are left alone.

The webhook deploy runs these steps in this order. The server does the first two itself, before anything on disk changes:

1. `git fetch origin`, and the commit `origin/main` now points at
2. the snapshot, through the server's own connection to the database
3. `git reset --hard <that commit>`
4. `npm install --omit=dev`
5. `scripts/preflight.js` against the snapshot, the check described in Section 5
6. `pm2 restart`

If a step fails, the deploy stops there and the server is not restarted. It keeps running the version it has already loaded, and new migrations never run without a fresh snapshot beside them. A failed fetch or snapshot leaves everything as it was. When a later step fails, the deploy puts back the commit the server is running (`git reset --hard` and `npm install`), so the files on disk match the running code again: the phones keep the frontend that goes with the API they talk to, and a crash or a reboot starts the version that was running, not the refused one. The restart step does not roll back. The next push to main tries again.

Only one deploy runs at a time. A push that arrives while one runs, or while an admin restores a backup, waits; only the newest waiting push is kept, because a deploy is always of `origin/main` as it is by then. It runs when the current deploy ends without a restart. When the current one succeeds and restarts the server, the waiting push is handed over in `data/deploy-pending.json`, and the restarted server deploys it only if `origin/main` has moved past the commit it now runs.

An admin can read the outcome in `/api/health/full` under `deploy`: `status` (`in_progress`, `success` or `failed`), for a failure the `step` it stopped at (`fetch`, `snapshot`, `prepare`, `update`, `install`, `preflight` or `restart`), the commit, the name of the snapshot that attempt took, whether a failed deploy put the running commit back (`rolledBack`), and the failed step's last lines of output (`detail`). The whole output of the latest attempt's chain is in `data/deploy.log`.

The webhook does not archive `data/photos`. A deploy never touches photo files, and migrations change only the database.

### Photos

Photos (contamination reports and lab-book experiments) are files under `data/photos/`, not in the database; the database only holds their path, size and SHA-256. A photo is written once under a random name and never changed, only deleted, which is what the photo backups rely on:

- **A second copy on the same disk, every night.** With the daily backup (at start-up and at midnight) every photo gets a hard link under `backups/photos/` (`scripts/photo-mirror.js`). A hard link is one file with two names, so this costs no space, and a photo deleted from `data/photos/` — by the app, by a person, by a bug — keeps its bytes under `backups/photos/`. Deleted photos stay there for 90 days, then go. Where no hard link can be made (`backups/` on another file system, FAT, some network shares) each photo is copied instead, once. The result of the last run is in `/api/health → backup.photos`.
- **The USB stick** (Section 10) copies each photo exactly once and is the copy that survives a dead disk.

`START.bat` and `update_server.sh` no longer archive `data/photos` on every run. Archives from earlier versions (`backups/photos_*.zip`, `backups/photos_*.tar.gz`) are left where they are, and they are not covered by the stick or the mirror: both copy what is in `data/photos` now. An archive can hold the only copy of a photo that is gone from there — the photos of a report deleted before the upgrade, or lab-book photos removed by the old "Start clean" while their records stayed — and an older database backup (`pre-reset-*.db` is never rotated) may still refer to it. So do not simply delete them. To free the space, copy them onto another medium (the stick, outside `MEISTERTRACKER-BACKUP/`, or an external disk) first, and keep at least the newest one.

### The lab history's hash chain

The lab notebook's history (who changed what, and when) is append-only in the database, and triggers stop anyone who tries to change or delete an entry through the app or SQLite. Someone holding the file can remove those triggers, so every entry also gets a link in a hash chain, written in the same transaction as the entry: the SHA-256 of the link before it and the entry itself (`db.js`, "Pruefkette"). An entry changed or deleted outside the app no longer matches its link, and one slipped in between others has no link and never gets one. Either way the place shows up:

- **Settings → Server** shows the chain as intact or names the first entry that no longer matches, with the newest timestamp and whether the stamps' signatures checked out.
- **`/api/health → labChain`** (admin) and **`/api/health/full → labChain`** give the same, with `broken` listing up to ten places and `signatures` counting the stamps by their signature (below); a broken chain or a bad signature turns the status to `degraded`.
- **The printed lab book** ends with the chain's state and its newest link, so a printout can later be held against the file, and says whether the newest stamp's signature was checked and with which certificate.

**Entries written without a link.** An older version of the app does not know the chain. If one runs against a database that has it, after a rollback or a revert across migration 97, its history entries go in at the end without links. The next start of a version that knows the chain, or its next history entry, links them in id order and writes an entry of its own that records which ones it took in (`entity` `chain`, `action` `adopt`). That is not reported as a break, but Settings → Server, `labChain.adopted` and every printed lab book name it from then on. Unlinked entries anywhere but at the end stay a break.

**The timestamp.** Someone who changes an entry and recomputes every link after it leaves a chain that matches itself. Once a day, if anything was added since, the server therefore asks a timestamp authority (RFC 3161) to sign the newest link: the request carries that SHA-256 value and a random number, nothing else. The signed token is kept in the database (`lab_timestamps`), and a rewritten chain no longer matches it. The server tries hourly until one succeeds; a failure shows under Settings → Server and in `labChain.stamping.lastError`. An admin can ask for one at once there. `LAB_TSA_URL` chooses the authority (Section 3); the default needs outbound HTTPS (port 443). The check reads the hash, the time and the serial number inside every stored token, not the columns next to it, and checks the authority's signature on it, since someone who can write the file could also store a token they made up. The time Settings → Server and the printed lab book show is the token's; a stamp whose columns say otherwise breaks the chain at its entry. What it cannot see is a stamp deleted together with its rows, since the stamps live in the same file; printed lab books, archives and the stick's snapshots keep earlier links and stamps outside it.

**The signature.** For every stored token the app checks what `openssl ts -verify` checks (`scripts/rfc3161-verify.js`): one signer, whose signed attributes cover the token's content by its hash and name the signing certificate; the signature over those attributes (RSA PKCS #1 v1.5, RSA-PSS, ECDSA or Ed25519); a certificate meant for time stamping (extended key usage `timeStamping`, critical, and no other usage openssl knows, as RFC 3161 asks); and a chain through the certificates in the token to a root certificate in Node's bundled list (`tls.rootCertificates`), every signature in it valid, every certificate below the root valid at the time the token names rather than today (the root is trusted as Node ships it), and no CA in it limited to other usages. So neither a certificate for something else, a web server's for instance, nor a CA that may only issue such certificates can make a token that passes. The last check goes beyond `openssl ts -verify`, which does not look at the CAs' usages. Each stamp gets one of three results, reported apart from the chain:

- `ok` — all of the above holds. `labChain.lastStamp.signer` names the certificate that signed the newest stamp.
- `bad` — something does not hold: the token was changed afterwards, or it never came from the authority. A token the app cannot check at all counts here too, one without the signing certificate (the app always asks for it) or signed with an algorithm the check does not cover (DSA or Ed448, say), since it cannot be told from a changed one. `labChain.signatures.problems` lists the first ten stamps that are not `ok`, with a reason. A bad signature turns the status to `degraded`.
- `unchecked` — everything that can be checked holds, but the app does not vouch for it: the chain ends at a root Node does not ship, it gets to one only through a CA limited to other usages, or the token holds up only with MD5, SHA-1 or another algorithm not on the check's list of strong ones. A change to the token's content, its signed attributes, its signature or the authority's own certificate never comes out `unchecked`: the digest and those signatures are checked first, whatever hash they name. A change to one of the CA certificates in the token can: once the chain no longer leads to a root Node ships, it looks like an authority whose root Node does not ship. So for an authority whose root Node does ship, the default one among them, an `unchecked` stamp deserves the same look as a `bad` one. It is no break; OpenSSL below checks such a token with the authority's root certificate.

The app does not check revocation (CRL or OCSP), nor which authority signed: any authority whose time stamping certificate chains to a root in Node's list passes, including one a CA under such a root has certified, which is why the signer is named. Node's list changes with Node updates, and a root it drops turns that authority's stamps from `ok` to `unchecked`. The default authority's tokens chain up to `USERTrust RSA Certification Authority`.

**A restored backup.** Restoring a backup (Section 11) brings back the backup's history and its chain, and the chain goes on from the backup's newest link. A printout or archive made before the restore can then name a link the file no longer has. The restore writes an entry that says so, with what it discarded and where that is kept; Settings → Server, `labChain.restores` and every printed lab book name it.

The chain starts when an installation first runs this version. What changed in the history before that cannot be shown by it.

To check a token without the app, for example years later, write it out with the sqlite3 shell and verify it with OpenSSL against the authority's root certificate (part of the usual CA bundles):

```bash
sqlite3 meistertracker.db "SELECT writefile('stamp-' || t.id || '.tsr', t.token), c.hash FROM lab_timestamps t JOIN lab_history_chain c ON c.history_id = t.history_id"
openssl ts -verify -digest <hash> -in stamp-1.tsr -token_in -CAfile /etc/ssl/cert.pem
```

Take the hash from the chain, as above, not from the stamp's own column. `Verification: OK` then means the authority signed exactly that link at the time the token names. OpenSSL checks the certificates as of today, so once the authority's certificate has expired, add `-attime` with the token's time in seconds since 1970 (on Linux, `date -u -d <genTime> +%s`). It then checks them at the time of the stamp, as the app does.

### The lab book archive

For a report, an audit or the yearly archive, an admin downloads the lab book as one ZIP file: under the experiment list, **Laborbuch drucken** → choose the period (and a work package) → **Archiv (ZIP)**. It holds what the printed lab book shows, as `laborbuch.json`, plus everything needed to check it years later without the app:

- `fotos/` — every photo of the selection as the app shows it, and the camera original where there is one;
- `pruefkette/` — the whole history and its hash chain, whatever the selection (a chain can only be recomputed from its first link);
- `zeitstempel/` — every RFC 3161 token and the link each one covers;
- `pruefung.json` — the chain's state, every stamp's signature as the app checked it (`stamps`), and every photo file checked against its recorded SHA-256 while the archive was written (a mismatch or a missing file is listed there);
- `SHA256SUMS` — every file's SHA-256, for `sha256sum -c SHA256SUMS`;
- `LIESMICH.txt` — the same in words, with the commands for each check.

The files are stored uncompressed (photos would not get smaller) and the archive uses ZIP64 once it passes 4 GB, which Windows Explorer, macOS and `unzip` read. The server builds one archive at a time and reads every photo twice, so a year with many originals takes a while. Keep the archive off the lab PC, like the stick.

### Verify Backup Health
```bash
# Quick check — exits 0 if healthy, 1 if degraded, 2 if critical.
node scripts/check-backup-health.js

# Full restore test of a specific file (uses the same openDb path as
# /api/backup/restore, so any schema-ordering bug surfaces here).
node scripts/verify-backup.js --restore-test backups/meisterpilze_backup_2026-04-29.db

# Full restore test of the most recent file in the backup dir — what an
# operator should run nightly from cron.
node scripts/verify-backup.js --latest

# Round-trip: take a fresh backup of the live DB and re-open it.
node scripts/verify-backup.js
```

## 10. Off-site backups (REQUIRED)

Backups stored on the same machine as the database are not enough — if the server fails, the backups go with it. On a PC someone can plug a stick into, the app does this itself (below). On a server elsewhere, use `rsync` to another host.

### The USB stick (built in)

**Setting up.** In **Settings → Backup → USB stick**, plug in a stick, press **Set up a stick**, pick its drive and give it a name. The app's own disk is not offered. The first stick creates the **recovery key**, 32 characters: put it in a password manager or print it, and keep it away from the PC. Without it, no stick can be read once the PC is gone. Until someone ticks that it has been put away, the key is shown again with every setup and the start screen asks for it, so a first setup that failed half-way cannot lose it. Further sticks use the same key; two sticks in rotation, one always somewhere else, is the arrangement to aim for. **Show key** displays it again for an admin.

**Every day.** Admins see a line on the start screen while today's backup is missing. Plugging in a stick of this installation is all it takes: the server notices it within 15 seconds, writes the backup by itself, and the line turns into "Backed up. The stick can come out." A stick that stays plugged in is backed up once a day. A run that failed for a reason a person has to fix (stick full or write-protected) is not retried that day while the stick stays in, and its reason stays on the start screen; pull the stick and plug it in again, or press **Back up now**. A stick left in gets one more try each day. One that may pass by itself (the stick unreachable for a moment, trouble on the PC) is tried again after ten minutes. A run that finds less than half the photos, or a database under a fifth of the size, of the last backup on that stick stops instead (a new PC before its restore, an emptied folder); **Back up anyway** goes ahead when that is meant, for example after **Start clean**. A photo the PC cannot read (a file another program holds, a folder the server's account may not open) is not on the stick; the run still finishes, but the start screen says the backup is not complete, the card lists what is missing, `/api/health → backup.usb.incomplete` is true, and the next run tries those photos again.

**What it writes**, below `MEISTERTRACKER-BACKUP/` on the stick:

| Path | What |
| --- | --- |
| `stick.json` | the stick's id, name, key id and key salt — plain text, nothing secret |
| `db/meistertracker_<number>_<UTC time>.enc` | a fresh database snapshot in the format **Settings → Backup → Restore** reads, encrypted with the recovery key; numbered in the order written, so a wrong PC clock cannot reorder them; the newest 30 stay, counted only when a run finishes — a run that fails after the database step leaves one extra snapshot, which the next run replaces, so retries cannot push the older ones out |
| `photos/<path>.enc` | every photo, AES-256-GCM, with its path and SHA-256 inside the encryption |
| `removed.json` | photos the app no longer has; they are deleted from the stick after 90 days |
| `last-backup.json` | what the last run did — plain text |

A photo already on the stick is never written again, so after the first run a backup takes seconds and the stick grows by the new photos only. Files are written under a temporary name, flushed and renamed, so a stick pulled half-way never holds a short file under a real name; the next run cleans up and continues.

The app keeps the key and the registered sticks in `data/usb-backup.json` (owner-only, never served), flushed to disk on every save, with the previous version beside it as `usb-backup.json.bak`. If the file does not read (cut off by a power failure), the copy is used; if neither reads, the app writes nothing until one does, so a new setup cannot replace the key — put the file back from a backup, or remove it and connect the old sticks with the recovery key. After each run it writes the marker file described below, so `/api/health` and `check-backup-health.js` report a stale stick backup like a stale off-site sync; `/api/health → backup.usb` says whether today is covered, and whether the last backup is complete.

`USB_BACKUP=off` in `.env` turns the stick backup off — the drive scanning and the start-screen reminder — for a server nobody can plug anything into. `USB_BACKUP_ROOTS` replaces the list of places a stick can show up (on Windows every drive letter from `C:`; elsewhere `/media`, `/run/media`, `/mnt`, `/Volumes`), separated like `PATH`.

### rsync to another host

The pattern for a server that is not within reach of a USB port is `rsync` over SSH to a separate host:

```bash
# /etc/cron.d/meisterpilze-offsite
30 0 * * * meisterpilze /usr/local/bin/meisterpilze-offsite-sync.sh
```

```bash
#!/bin/bash
# /usr/local/bin/meisterpilze-offsite-sync.sh
set -e
SRC=/var/www/meistertracker/backups/
DST=backup-host:/srv/meisterpilze-offsite/
TARGET="backup-host:/srv/meisterpilze-offsite/"
START=$(date +%s)
rsync -az --delete "$SRC" "$DST"
END=$(date +%s)
BYTES=$(du -bs "$SRC" | awk '{print $1}')
# Write a marker so /api/health and check-backup-health.js know this
# script ran. Without the marker, off-site failure is silent.
cat > /var/www/meistertracker/backups/.offsite-sync.json <<EOF
{ "time": "$(date -u +%Y-%m-%dT%H:%M:%SZ)", "bytes": ${BYTES}, "target": "${TARGET}", "durationSeconds": $((END-START)) }
EOF
```

### SSH key for the cron user
```bash
sudo -u meisterpilze ssh-keygen -t ed25519 -f /home/meisterpilze/.ssh/id_ed25519 -N ""
sudo -u meisterpilze ssh-copy-id backup-host
sudo -u meisterpilze ssh backup-host true   # confirm passwordless login works
```

### Marker file
The marker (`backups/.offsite-sync.json`) lets the server expose `/api/health → backup.offSite` with `lastSync`, `ageMinutes`, and `target`, and lets `check-backup-health.js` exit 1 (degraded) when the off-site sync hasn't run in `--max-offsite-age-hours` (default: 26). Any cron script in any language can update the marker — see the bash example above.

### Alternative: OneDrive on Windows
If the project lives under a OneDrive-synced folder, the `backups/` directory cloud-syncs automatically. This is convenient but **platform-specific and not portable** — on a Linux server you must use rsync (or another off-site tool that touches the marker file).

## 11. Restoring from a backup

Three scenarios in increasing severity, then the stick and a single photo:

### A. Encrypted `.enc` restore via the admin UI

If you downloaded a password-encrypted archive from **Settings → Backup → Download**, restore it the same way:

1. Log in as admin.
2. Open **Settings → Backup → Restore** and pick the `.enc` file.
3. Enter the password used at download time.
4. The server validates the file, copies the current database first, swaps in the new one, and broadcasts the change to all connected clients. If the new database does not open, it goes back to the copy.

The copy of the database as it was before the restore is kept, not deleted: `backups/pre-restore-<time>.db`, never rotated, like the copy taken before a reset. It is taken with `VACUUM INTO`, so it is complete even while another program reads the file. It holds whatever the restore discarded, the lab notebook's history entries and timestamps among them. Delete old ones by hand once nobody needs them.

The restore is recorded in the lab notebook's history, in two ways:

- **One entry for the whole restore** (`entity` `database`, `action` `backup`, no experiment), written even when the backup holds no experiment and no work package. It names the kept copy, the newest history entry the backup has (`entity_id`), how many history entries the restore discarded, and how many of them belonged to each experiment and work package. That includes experiments and packages created after the backup, which the restored database no longer has; they are marked as not in the backup. One counts as in the backup when the backup holds one of its history entries, or one under the same id that is the same one: for an experiment the same code, for a work package, whose code can change, the same creation time. The id alone is not enough, since after an earlier restore it can belong to another experiment or package. Nor is the code: when the backup lacks the entry an experiment or package was created with, it came after the backup, even if it got the same id and code as the one the backup has. Settings → Server shows the latest restore, and every printed lab book lists all of them under its hash chain, as does `labChain.restores` in `/api/health`. The lab book archive has the entry in `pruefkette/verlauf.json` like any other, and `LIESMICH.txt` explains it.
- **One entry in every experiment and work package** of the restored database, with that one's own count and the name of the kept copy.

What counts as discarded is every entry of the history before the restore that the backup does not hold. The server reads the backup as it is, before its migrations run, and compares each entry's link in the hash chain (Section 9) with the backup's link under the same id. A link covers its entry and everything before it, so the count stays right when ids were handed out a second time after an earlier restore: restoring the morning's backup and then the noon one discards entries whose ids the noon backup also has, but with other content. Where one side has no link under an id, the entry itself is compared. A backup from before the hash chain (migration 97) has no links, so there every entry newer than the backup's newest one counts. After an earlier restore that can be too few, and the printed lab book says it was counted that way. If a backup cannot be read as it is, the server counts by id against the opened file instead, records that (`counted` `migrated`), and the printed lab book says so too.

The entry an earlier restore wrote is part of the history like any other. When a later restore discards it, it is in that later restore's kept copy, which therefore leads on to the copy before it.

The restore is refused rather than risked in two cases: when that copy cannot be written (disk full), and when another program still has the database open after the server closed it (the DuckDNS fallback task reads it every five minutes, or a database tool). An open database leaves its write-ahead log behind, and SQLite would replay it into the restored file. Try again a minute later.

If the restored file does not open, the server puts the copy back and opens that. If even that fails, it says so and keeps the database closed rather than run on a file nobody chose. The answer says whether the database file is the data from before again or still the restored backup, and names the copy; in the second case copy it over the database file before restarting, or the server comes back on the backup.

This is the preferred path when the server is still reachable.

### B. Manual file swap (admin UI unreachable)

When the database is corrupt or the admin UI can't load:

```bash
# 1. Stop the server
pm2 stop meisterpilze        # Linux
# Or close START.bat            # Windows

# 2. Move the broken DB aside (KEEP it — never delete a corrupt DB; you may
#    need it for forensics or partial recovery later)
mv meistertracker.db meistertracker.db.broken
rm -f meistertracker.db-wal meistertracker.db-shm

# 3. Pick a backup. Auto-backups: backups/meisterpilze_backup_YYYY-MM-DD.db
#    Snapshots from before an update (the newest 10, and any that predate
#    the rotation):
#    backups/meistertracker_YYYYMMDD_HHMMSS.db
ls -la backups/

# 4. Verify the chosen file BEFORE swapping it in (catches a corrupt file
#    early, avoids a second outage on restart):
node scripts/verify-backup.js --restore-test backups/meisterpilze_backup_2026-04-29.db

# 5. Copy the backup into place
cp backups/meisterpilze_backup_2026-04-29.db meistertracker.db

# 6. Restart
pm2 start meisterpilze       # Linux
# Or double-click START.bat     # Windows
```

After restart, log in and confirm batches/harvests/users are present. If anything is missing, stop again and try an older backup.

### C. WAL-only recovery (last resort)

If `meistertracker.db` is gone but `meistertracker.db-wal` and `meistertracker.db-shm` survive, **you cannot recover from WAL alone** — SQLite's write-ahead log is meaningless without the original DB file it amends. Treat this as scenario B and accept the loss of whatever was sitting in WAL since the last backup.

To minimise this loss in the future, lower `wal_autocheckpoint` (currently 1000 pages) or run `PRAGMA wal_checkpoint(TRUNCATE)` from a cron, but **the off-site backup (Section 10) is the real defence** — it captures the consistent on-disk state every night.

### D. From the USB stick

You need the stick and the recovery key. Run the commands below in the app's folder, in the Windows command prompt or PowerShell, with the stick in `E:` (change the letter to the stick's). On Linux, put the stick's mount point (for example `/media/<user>/<stick>`) where `E:\` stands. Check the stick first; this reads and decrypts everything and writes nothing:

```bat
node scripts/usb-restore.js E:\ --verify
```

The key is asked for, or read from `USB_BACKUP_KEY`.

**The database**, with the server running: **Settings → Backup → Restore**, pick a file from `MEISTERTRACKER-BACKUP/db/` on the stick (the highest number is the newest), and type the recovery key as the password. With no running app — a new PC — stop the server and write it into place instead:

The newest snapshot:

```bat
node scripts/usb-restore.js E:\ --db meistertracker.db
```

An older one, named as it is in `db/` on the stick, replacing a database that is already there:

```bat
node scripts/usb-restore.js E:\ --db meistertracker.db --from meistertracker_000012_20261001T090000Z.enc --force
```

`--force` moves an existing `meistertracker.db` and its `-wal` and `-shm` files aside (renamed with `.before-restore-<time>`, never deleted). They have to go together: a log left beside the restored file would be replayed onto it. The snapshot is checked with SQLite's quick check, and opened on a copy the way the server opens its database (schema and migrations), before anything is moved, and if the database is still in use, everything is put back and the script says so.

**The photos**: missing ones go back to their old paths, byte for byte; existing ones are left alone unless `--overwrite`:

```bat
node scripts/usb-restore.js E:\ --photos data/photos
```

A photo whose checksum or name does not match is reported and skipped. On a new PC, restore first, then open **Settings → Backup**: the stick is offered as "a stick from an earlier installation", to be connected with its key. Connecting writes nothing to it: the first backup onto it waits until someone presses **Back up now**, so a stick connected before the restore cannot get the empty database as its newest snapshot. From then on it is backed up every day by itself.

### E. A deleted photo

A photo deleted in the last 90 days is still under `backups/photos/` with the same path it had under `data/photos/`. Copy it back; on Windows:

```bat
copy backups\photos\2026\10\exp-4\0b1c2d3e.jpg data\photos\2026\10\exp-4\
```

On Linux (`-n` leaves a file that is there alone):

```bash
cp -n backups/photos/2026/10/exp-4/0b1c2d3e.jpg data/photos/2026/10/exp-4/
```

The database still has to know the photo; for a photo whose record is gone too, restore the database first (A, B or D).

For background on the encrypted-archive format and the admin restore endpoint, see README.md → Restoring from a backup.

## 12. Label Printing on Linux

The label-printing endpoint (`/api/print`) needs the Windows print spooler to talk to a Zebra GK420d. On Linux you have two practical options:

### Option A — ZPL download fallback (works out of the box, no setup)

If `PRINT_BRIDGE_URL` is unset, every print button on the Linux server falls back to producing a ZPL file the browser downloads. The user sends the file to a Windows PC that has the Zebra driver and prints it from there (double-clicking the `.zpl` typically works).

Pros: zero setup, works from any browser device.
Cons: extra click per print, no live printer-status indication.

### Option B — Windows print bridge (recommended for daily lab use)

Run [`scripts/print-bridge.ps1`](scripts/print-bridge.ps1) on a Windows PC that has the Zebra GK420d attached. The Linux server forwards `/api/print` and `/api/printer-status` calls to the bridge over **HTTPS** (with a self-signed cert the installer generates automatically), so print buttons go straight to the printer like the Windows-native install used to and the LAN traffic is encrypted.

The Print tab's status banner reflects the bridge state in real time:
- **Green** "Printer ready" — bridge reachable, printer online
- **Yellow** "Printer disconnected" — bridge reachable, but the GK420d is unplugged or off
- **Red** "Print bridge unreachable" — Windows PC off or service not started
- **Blue** "ZPL download mode" — no bridge configured, buttons download instead

#### One-time Windows setup

The script ships with a self-installer that handles the TLS certificate, URL ACL, firewall rule, scheduled task, and immediate start in one step.

1. **Download** `print-bridge.ps1` from the running server's **Settings → Drucker** tab (or directly from `scripts/print-bridge.ps1` in the repo) and save it to e.g. `C:\meistertracker-bridge\print-bridge.ps1`.

2. **Install** by running this from any PowerShell (the installer auto-elevates via UAC if needed):
   ```powershell
   powershell -ExecutionPolicy Bypass -File "C:\meistertracker-bridge\print-bridge.ps1" -Install
   ```

   The installer:
   - Generates a self-signed TLS cert (10-year validity) into `cert:\LocalMachine\My`
   - Binds it to the listener port via `netsh http add sslcert`
   - Adds the HTTPS URL ACL and the inbound firewall rule
   - Registers the "MeisterTracker Print Bridge" scheduled task (At Logon)
   - Starts the task immediately

That's it — the bridge is now serving HTTPS on port 9100 and will start automatically at every logon.

#### Management commands

```powershell
# What's installed and running?
print-bridge.ps1 -Status

# Stop the bridge but keep it installed (Settings → Drucker on the server can still
# show "configured" — switch the server to local PowerShell printing while
# you do hardware maintenance, etc.)
print-bridge.ps1 -Disable
print-bridge.ps1 -Enable

# Remove TLS cert, SSL binding, URL ACL, firewall rule, scheduled task,
# and stop any running instance
print-bridge.ps1 -Uninstall
```

All four commands auto-elevate.

#### Server-side configuration

The Linux server's **Settings → Drucker** tab is the recommended place to enter the bridge URL + token — values are stored in the database and take effect immediately, no server restart needed.

For headless deployments or backwards-compatibility, the same values can also be set in `.env` (UI values take precedence when present):
```ini
PRINT_BRIDGE_URL=https://<windows-pc-ip>:9100
PRINT_BRIDGE_TOKEN=<a-long-random-string>
```

#### Token auth (optional, recommended)

The HTTPS channel encrypts the token in transit, but token auth is still recommended on shared LANs to prevent unauthorized prints. Without a token, anyone who can reach `https://<windows-ip>:9100` can print to your Zebra. To require a token, set the same value on both sides.

On the Linux server: enter the token in **Settings → Drucker** (or `PRINT_BRIDGE_TOKEN` in `.env`).

On Windows: pass `-Token` when installing, e.g.:
```powershell
powershell -ExecutionPolicy Bypass -File "C:\meistertracker-bridge\print-bridge.ps1" -Install -Token "your-long-random-string"
```

The installer persists the token into the scheduled-task arguments, so it survives logoffs / reboots.

#### App passwords for calendar clients

CalDAV authenticates with HTTP Basic against the app's own accounts, so
subscribing a phone used to mean typing the password that also opens the web UI
into iOS or Thunderbird — where the client keeps it, in a keychain that syncs to
a cloud backup and outlives the device.

Under **Settings → CalDAV** each user can now create an app password per device.
It opens calendars and nothing else, it is shown once at creation and stored
only as a hash, and it can be revoked on its own without changing the account
password. The account password still works, so existing subscriptions keep
running — but there is no longer a reason to hand one to a calendar client.

Changing or resetting an account password deletes that user's app passwords,
along with their sessions and OAuth grants. A password change is the answer to
"this account may be compromised", and somebody who had the account could have
created one; the cost is re-adding the calendar on each device.

#### Certificate pinning

The bridge's certificate is self-signed, so there is no chain for the server to
validate. Instead it pins the certificate: the **first** connection to a bridge
address records that certificate's SHA-256 fingerprint (you will see
`Pinned print bridge certificate on first connection` in the log, with the
fingerprint), and every connection after that is refused unless the certificate
matches. That is what stops somebody on the same LAN from answering for the
bridge's address, collecting your `PRINT_BRIDGE_TOKEN` and printing whatever
they like.

Consequence: **re-running `print-bridge.ps1 -Install` issues a new certificate**,
and printing will then fail with *"Bridge certificate changed"* until you tell
the server to trust it. To re-pin, open **Settings → Drucker** and save — any
save clears the pin, and the next print records the new certificate. Pointing
the server at a different bridge address re-pins by itself; only a changed
certificate at the *same* address is treated as a problem.

If you see that error and you did not just re-install the bridge, do not save
the settings — find out who else is answering on that address first.

## 13. Updating

To update a running installation:
```bash
cd /var/www/meistertracker
bash update_server.sh
```

The script runs six steps, numbered like the `[n/6]` lines it prints:
1. Fetch from GitHub and `git reset --hard origin/main`, which discards local changes to tracked files. Skipped when the script runs from a git worktree.
2. Install dependencies (`npm install --production`).
3. Snapshot the database into `backups/` as `meistertracker_YYYYMMDD_HHMMSS.db` (Section 9).
4. Generate a self-signed TLS certificate if `certs/server.key` or `certs/server.crt` is missing (Section 4).
5. Check the new version against a copy of the database with `scripts/preflight.js`. If that fails, the update stops here and the old server keeps running (Section 5).
6. Restart the server via PM2 and check that it stayed up. If it did, the commit is tagged `stable`; if it crashed, the script rolls back to `stable`, the last commit that started (Section 5).

On a server that uses DuckDNS, a successful update ends with a reminder if the DuckDNS fallback timer is not running (Section 6).

## 14. Docker Deployment (Alternative)

MeisterTracker includes a Dockerfile for containerized deployment:

```bash
docker build -t meistertracker .
docker run -d \
  --name meistertracker \
  -p 3000:3000 \
  -v meistertracker-data:/app/meistertracker.db \
  -v meistertracker-backups:/app/backups \
  -v meistertracker-calendars:/app/calendars \
  meistertracker
```

The container uses Node.js 22 Alpine, runs as a non-root user, and includes a health check on `/api/health`.

## 15. Outbound Harvest Feed (Optional)

Pushes a signed summary of recorded harvests and upcoming batches to a URL you choose, so a shop, listing page or chat bot can answer "what's available today?" without this machine being reachable from the internet. See **README.md → Harvest feed** for the payload shape and what it deliberately leaves out.

**Nothing is opened up.** No inbound endpoint, no port, no dependency on your public IP. The server dials out; that is the whole surface.

The receiver may, however, answer — the reply can carry pickups back, and withdraw them again, which lands on the **Pickups** page. Data from the far end therefore does reach the database, over a socket this side opened and is still holding. The body is validated as untrusted input (JSON only, 64 KB cap applied before it is read, every field bounded, unknown fields dropped), and a malformed reply never turns a delivered push into a failed one. See **README.md → Pickups** for the shape.

### Setup — in the browser

**Settings → Harvest feed.** No shell, no file, no restart.

1. **Receiver URL.** HTTPS; plain HTTP is accepted for `localhost` only, so you can try it against a local receiver first.
2. **Generate** a secret and give the same value to the receiving side. It is stored write-only — the page never reads it back, so copy it before saving.
3. **Show what would be sent.** This builds the payload from the database and displays it, sending nothing. Do this before enabling: "does this leak anything?" is a question to answer by looking, not by trusting the description.
4. **Save**, then **Send one now.** The result says whether the receiver accepted it, which is the difference between saved and working — a wrong secret or a typo in the host looks exactly like a correct setup until something is actually delivered.

The timer restarts on save. The last outcome stays visible at the top of the page, so a feed that quietly stopped delivering shows up instead of failing in silence.

### If a shop sells from this

The feed never reports a harvest total as sellable. What a shop may sell is always a **release**: an amount somebody set aside by hand, on the **Pickups** page, per species, with a date it stops counting. Put that amount in its own crate and sell everything else from the rest — then a walk-in customer or a busy market Saturday cannot make the published number wrong, and there is nothing to keep up to date between harvests.

The harvest totals still travel, in `harvested`. They are production data — what came off the racks — and a receiver that publishes them as stock is reading the wrong field.

Two consequences worth knowing:

- **The payload is version 2.** A receiver built for version 1 should reject it, because ignoring the release list means publishing produce you kept back. Check the receiver before pointing this at one.
- **Nothing is released until you say so.** A fresh install reports an empty release list, and a shop reading it shows nothing. That is the safe direction: an empty list is a statement, not a gap.

**In what portions it is sold** is the second half of the same question, and it is answered in *Settings → Harvest feed → Amounts handed out*: tick the sizes you pack in — 250 g, 500 g, a kilo — or add one the list does not offer. They go out as `packSizes`, one list for every species, because portioning follows the packing bench and not the mushroom. A shop offers those amounts for each listing and leaves out whatever the release no longer covers.

Ticking nothing leaves the field out of the payload entirely, and a receiver then has no portions to sell produce in. ⚠️ **Expect that to mean no orders at all** — a shop built for this feed shows the listing and its price, and offers no order button until sizes are set. It deliberately does not fall back on sizes of its own: a ladder nobody decided looks, in a shop window, exactly like one somebody did.

> **Until 2026-08-14 this was a switch** (`HARVEST_WEBHOOK_RELEASE_MODE`, "Only report what is released for sale"), off by default. It is gone. Off, the feed published harvest totals — a number that stops being true the moment something is sold at a stall — and because it was also the quiet option, a lab that never found the checkbox published raw stock and looked fine doing it. A setting whose wrong value produces plausible-looking numbers is not a setting worth keeping. The database column stays and is always 1, so an older build reading the same file does not fall back.

### Setup — from a file instead

For installs where configuration is baked into an image or handled by whatever starts the process, the same settings exist as environment variables. They apply whenever the stored config is **off**; enabling it in Settings takes over, and the page says which of the two is in charge. The two are never merged — a URL from one place and a secret from the other is a configuration nobody can read off a single screen.

1. **Generate a secret** and give the same value to the receiving side:

   ```bash
   openssl rand -hex 32
   ```

2. **Add both values to `.env`:**

   ```ini
   HARVEST_WEBHOOK_URL=https://example.org/harvest
   HARVEST_WEBHOOK_SECRET=<the value from step 1>
   ```

3. **Look at the payload before you turn it on.** This reads the database and prints what would be sent, without sending it:

   ```bash
   node harvest-feed.js --dry-run
   ```

4. **Send one for real**, to check the receiver accepts the signature:

   ```bash
   node harvest-feed.js --once
   ```

5. **Restart the server.** The feed then runs on its own timer.

   ```bash
   bash update_server.sh
   ```

### Tuning

| Variable | Default | What it does |
|---|---|---|
| `HARVEST_WEBHOOK_URL` | — | Where to POST. Unset means the feature is off. HTTPS required (plain HTTP only for `localhost`, so you can try it against a local receiver). |
| `HARVEST_WEBHOOK_SECRET` | — | HMAC key. Required — the feed refuses to start without one. |
| `HARVEST_WEBHOOK_INTERVAL_MIN` | `15` | How often to post. Minimum 1. |
| `HARVEST_WEBHOOK_FRESH_DAYS` | `3` | How far back a harvest still counts as on offer. Set it to what your product actually keeps. |
| `HARVEST_WEBHOOK_PLANNED_DAYS` | `28` | How far ahead to report upcoming block batches. `0` drops the planned block entirely. |
| `HARVEST_WEBHOOK_LEAD_DAYS` | `0` | Days between a batch's due date (end of incubation) and the first expected flush. Species-dependent — yours is whatever your records show. |
| `HARVEST_WEBHOOK_STRAIN` | `1` | `0` sends species only, no strain names. |
| `HARVEST_WEBHOOK_SITE` | — | Free-form label, passed through untouched. Useful when several sites post to one receiver. |
| `HARVEST_WEBHOOK_PACK_SIZES` | — | The portions a release is handed out in, in grams: `250,500,1000`. One list for every species. Whole numbers from 25 to 25000, at most eight; anything else in the list is dropped. Unset means the field is absent — and a receiving shop has nothing to sell produce in, so expect no orders until it is set. |
| `HARVEST_WEBHOOK_TIMEOUT_MS` | `15000` | Per attempt. Three attempts with backoff; a 4xx other than 408/429 stops immediately, because a wrong secret does not fix itself. |

### Verifying the signature on the receiving side

```js
const crypto = require('crypto');
// `raw` must be the exact request body, before any JSON parsing.
const ts = Number(req.headers['x-meistertracker-timestamp']);
if (!Number.isFinite(ts) || Math.abs(Date.now() / 1000 - ts) > 300) reject(); // replay window
const expected = 'sha256=' + crypto.createHmac('sha256', SECRET).update(`${ts}.${raw}`).digest('hex');
const got = String(req.headers['x-meistertracker-signature'] || '');
// Compare in constant time, and check the length first — timingSafeEqual throws
// on a length mismatch, and the caller controls that header.
const ok = got.length === expected.length && crypto.timingSafeEqual(Buffer.from(got), Buffer.from(expected));
```

The timestamp check is the half that stops replays; without it a captured request stays valid forever.

### Troubleshooting

| Symptom | Cause |
|---|---|
| `Harvest feed misconfigured — not started` in the log | The URL is set but the secret is not, or the URL is not HTTPS. The message names which. |
| `Harvest feed skipped {reason: worktree mode}` | `WORKTREE_MODE=1`. Deliberate — two servers posting contradictory snapshots to one receiver is worse than one posting none. |
| Receiver gets a 401 back from your own check | Verify over the **raw** body, not a re-serialized object; `JSON.stringify` of a parsed payload will not byte-match. |
| `harvested` is empty but you harvested today | Check `HARVEST_WEBHOOK_FRESH_DAYS`, and that the harvest rows carry a species. |
| `planned` is empty | Only `block` batches with bags, a due date inside the window, and no harvest recorded yet appear there. |
| `Harvest feed reply not usable` in the log | The receiver answered, and the body did not survive validation. The message says why — `reply is text/html, not JSON` is usually a proxy in front of the receiver returning an error page. *Send one now* in Settings shows the same detail. |
| The Pickups page stays empty although the receiver sends some | Look at *Send one now*: it reports both the reply error and how many entries were dropped. Dropped entries usually mean no usable `id`, or a `from`/`to` carrying an offset — those must be local wall-clock, with the zone in `zone`. |
| The receiver keeps resending a pickup it was told about | It is only confirmed after a push that succeeded. Check the feed is actually delivering; a failed push deliberately confirms nothing. |
| A cancelled pickup is still on the Pickups page | The receiver has to name it in `pickupsCancelled` — there is no route into this machine for it to delete anything. It should repeat that until the id comes back in `pickupsDone`. |

## Quick Reference

| Command | Description |
|---------|-------------|
| `bash update_server.sh` | Update & restart |
| `bash update_server.sh start` | Start server |
| `bash update_server.sh stop` | Stop server |
| `bash update_server.sh status` | PM2 process status |
| `pm2 logs meisterpilze` | View server logs |
| `pm2 monit` | Real-time monitoring dashboard |
| `curl -k https://localhost:3000/api/health` | Health check |