# External monitoring — the probe that watches the node

Status: **required before the Cloud launch** (the launch gate warns when it
is missing).
Scope: DOMINUS Cloud — the managed offering. The community edition is
self-hosted and has no SLA.

## The gap this closes

The production stack runs Prometheus, Alertmanager and Grafana on the app
node (`deploy/prometheus/`, `deploy/terraform/modules/app-node/`). That is
the correct place to watch the *application* — provider error rates, stuck
queues, dead letters, DNS health — and the wrong place to discover that the
*machine* is gone.

When the app node fails, the node that would have reported the failure
fails with it. The alerts about the outage are generated by the outage.
The SLA in [sla.md](sla.md) promises 99.5% monthly availability, measured by
"an external probe, configured per deployment" — until that probe exists,
that number is an intention, not a measurement.

The specification of the probe is in
[`deploy/monitoring/uptime-probe.yml`](../../deploy/monitoring/uptime-probe.yml).
It is provider-neutral: the operator picks a free tier and translates it.

## What to configure

1. **A monitor** (UptimeRobot, Better Stack, Hetrix Tools, Checkly — the
   free tiers are enough for the launch topology) pointed at
   `https://<domain>/api/health`:
   - `GET`, expect `200`;
   - the body must contain `"status":"ok"` — a proxy answering `200` with an
     error page is a failure, not a pass;
   - a 60-second interval, a 20-second timeout;
   - at least one probe outside the app node's failure domain (any
     provider qualifies: the probe runs on the provider's infrastructure,
     not on yours).
2. **A dead-man's switch**: a scheduled ping to the monitor's heartbeat URL
   from a host that is not the app node. If the ping stops arriving, the
   monitor alerts. This is the only mechanism that catches the failure the
   probe cannot: the probe service itself is down.
3. **A notification channel** that is the same destination the Alertmanager
   webhook points at — and, ideally, a *different vendor* from the one
   hosting the monitor. Two alerting paths on the same provider are one
   alerting path.

## Why the health endpoint and not a feature route

The probe must not depend on an upstream service. If it polled a route that
resolves a domain, every USPTO, EUIPO, RDAP or DNS hiccup would read as
downtime and the availability figure would measure other people's
infrastructure. `/api/health` reports the process and its database and
nothing else: verdicts degrade, correctness does not
([ADR-0002](../../docs/adr/0002-scoring-engine-design.md),
[ADR-0039](../../docs/adr/0039-dns-consensus-degradation-policy.md)).

## The configuration variable

`EXTERNAL_PROBE_URL` is the operator's public health URL. It is read by the
launch gate (`npm run launch:check`), not by the server: the server has no
use for the URL of a monitor that watches it. It is documented in
`.env.example` alongside the other operator variables.

```
# The operator's external availability probe (docs/operations/external-monitoring.md).
# The launch gate warns when it is unset: without it, the SLA's availability
# figure is not measured.
# EXTERNAL_PROBE_URL=https://dominus.example/api/health
```

## What the launch gate says

`npm run launch:check` reports:

```
[WARN] external monitoring: EXTERNAL_PROBE_URL is not set — the app node's own
       monitor cannot report the app node's outage
```

It is a warning and not a failure because a deployment can be a staging
one. It is on the paid list as a failure: a paid customer is promised a
measured availability, and the measurement has to exist.

## The other half: the dead-letter and backup paths

The probe watches the API. Two scheduled paths can die silently and are
covered by alerts rather than by availability:

- the scheduler (renewal checks, revenue ingestion, `pg_dump` backups) —
  `SchedulerDown` in [`deploy/prometheus/rules.yml`](../../deploy/prometheus/rules.yml);
- the PITR base backup — `BackupStale` and `PitrBaseBackupStale` in the
  same file.

Both alerts go to the Alertmanager webhook. If the webhook is the
placeholder that ships in the repository, they alert nowhere: the launch
gate reports that as a warning too, and `terraform apply` substitutes the
real URL from `alertmanager_webhook_url`.

## What is NOT covered

- The availability of the **upstream data sources** (RDAP, WHOIS, USPTO,
  EUIPO, DNS roots, Stripe). The SLA excludes them explicitly: a degraded
  provider reduces the number of verdicts, not their correctness.
- The **private network** between the app node and the database node. A
  partition shows up as an unhealthy API, which the probe reports.
- **Restore correctness**. The probe says the service is up; only a restore
  drill proves the data comes back
  ([rto-rpo.md](rto-rpo.md)).