# dead-mans-switch A freshness checker for scheduled agent jobs. It does not watch for errors. It watches for the absence of success, and says so before you notice on your own. ## The problem A scheduled job that stops firing does not announce itself. Nothing throws, nothing pages, no log fills up with stack traces. The job simply stops running, and the report, update, or sync it was supposed to produce is quietly missing from where you would expect it. This happens more often than error-driven monitoring catches. A cron entry survives a server migration but the script it calls does not. A credential the job depends on expires and the job exits cleanly before doing anything, because nothing forced it to check. An agent's own recurring task gets dropped without a trace when its scheduler config changes. One lane behind this tool went dark for five weeks, unnoticed. No errors. No alerts. Just an increasingly stale set of outputs nobody was checking day to day. Catching more failure modes does not fix this. Checking for success directly does. ## How it works Each job you track writes a plain success string (a sentinel) into its own log file when it finishes. `deadmans.py` scans your log directory for each task's most recent log, inside a staleness window you configure, and checks for that sentinel on a line of its own. A task is a finding if: - no log exists yet, and the task is not flagged manual - logs exist, but none has a name the checker can read a run time from - the most recent log is older than its configured window - the most recent log has no success sentinel - the most recent log carries a failure sentinel, or a line matching a failure pattern, instead - the most recent log shows the job starting but never finishing The exit code inverts what you would expect from a linter. 0 means every tracked task is fresh. 1 means at least one needs attention. That makes `deadmans.py check` a one-line addition to a cron job or a CI pipeline. ## Install Install from source. ```bash git clone https://github.com/jimy-r/dead-mans-switch.git cd dead-mans-switch pip install . ``` This adds a `dead-mans-switch` command to your PATH. Every example below also works as `dead-mans-switch check` in place of `python deadmans.py check`. Installing is not required. `deadmans.py` is a single stdlib-only file, so cloning the repo and running `python deadmans.py` works exactly the same, with nothing to add to a scheduling environment beyond the file itself. ## Quickstart ```bash python deadmans.py init # writes a starter deadmans.json ``` Edit `deadmans.json` for your own tasks, then make sure each job writes its sentinel string into a log file that matches the configured pattern when it finishes successfully. ```bash echo "MY_JOB_OK" >> "logs/my-job_$(date +%Y-%m-%d)-$(date +%H%M).log" ``` Run it from cron, from CI, or from wherever your own scheduling already lives. ```bash python deadmans.py check ``` Exit code 1 means read the output. Exit code 0 means nothing to do. ## Config reference Top-level keys in `deadmans.json`. | Key | Meaning | Default | |---|---|---| | `log_dir` | Directory the tool scans for logs, resolved relative to the config file if not absolute | `logs` | | `log_pattern` | Filename template using `{task}`, `{date}`, `{time}` placeholders. `{date}` is `YYYY-MM-DD` and `{time}` is `HHMM` or `HHMMSS` | `{task}_{date}-{time}.log` | | `timezone` | Which clock wrote the `{date}`/`{time}` in your log filenames. `local`, `UTC`, a fixed offset like `+10:00`, or an IANA name like `Australia/Brisbane` | `local` | | `tasks` | List of tracked task objects, see below | required | Keys inside each task object. | Key | Meaning | Default | |---|---|---| | `name` | Task identifier, matched against the `{task}` slot in `log_pattern` | required | | `max_age_hours` | How old the most recent log can be before it counts as stale | required | | `sentinel` | Success string to look for in the most recent log | required | | `failure_sentinel` | Optional string that marks an explicit failure | none | | `failure_patterns` | Optional list of regular expressions. A log line that matches one marks a failure the same way `failure_sentinel` does, see below | none | | `start_sentinel` | Optional string the job writes when it *begins*, so a run that starts and never finishes reports `HUNG` rather than `NO_SENTINEL` | none | | `max_runtime_hours` | How long a started run may go without finishing before it counts as hung. Needs `start_sentinel` | none | | `line_stamp` | If true, one ISO 8601 time may sit in front of a sentinel, for a job whose logger stamps every line, see below | `false` | | `manual` | If true, a task with no log yet reports `MANUAL_OK` instead of a finding | `false` | | `artefact` | Optional second freshness signal keyed to what the job *produces* rather than to its log, see below | none | A key that is not in these two tables is a config error. `check` exits 2 and names it, so a misspelling such as `artifact` for `artefact` stops the run instead of quietly switching that setting off. See [`deadmans.example.json`](deadmans.example.json) for a working two-task example. ## Timezones A log filename carries no offset. `nightly-report_2026-01-15-0000.log` is 9pm or 10am depending on who wrote it, and the checker has no way to tell from the name alone. If your job stamps its filenames in UTC and you run the check from Brisbane, every task reads ten hours fresher than it is, which is the wrong direction for a tool whose whole job is catching things that stopped. Set `timezone` to whichever clock the *producer* uses. `local` (the default) is correct when the job and the checker run on the same host. `UTC` is the common answer for anything containerised or CI-driven. An IANA name needs a tz database on the machine, which ships with most Linux and macOS installs and comes from the `tzdata` package on Windows; an unresolvable name is a config error rather than a silent fall back to the wrong clock. One caveat on `local`: the host offset is read once per run, so a config left on `local` across a daylight saving boundary is an hour out until the next run. For up to an hour after the clocks go back, a log stamped just before the change also reads as ahead of the clock and is skipped (see the next section), so the check reads the run before it instead. Name the zone if that matters to you. ## Log names the checker skips The run time comes from the log's filename, and the newest log is the one with the latest time in its name. A name counts only when that time is real and not ahead of the clock. Three kinds of name are skipped. - A date or time that does not exist, such as month 13 or hour 25. - A stamp more than five minutes ahead of the checker's clock. The five minutes allow for two hosts whose clocks differ a little. - A `{time}` that is neither four digits nor six. Taking the last name in sort order would be simpler, and it is wrong. One stray file called `nightly-report_2099-12-01-0000.log` that holds the success line would report `FRESH` for as long as it sat in the directory, with every real log behind it unread and the job possibly dead. Each skipped name appears in the task's detail line, so a stray file is easy to find and delete. A task whose logs were all skipped reports `LOG_UNREADABLE`. That usually means the producer and the config disagree about the filename format or about the clock. ## The manual flag Not every tracked task runs on a fixed clock. Some get invoked by hand, or by an agent, on an irregular cadence. A periodic audit, a cleanup pass, something run when someone gets to it rather than on a schedule. For those, a missing log on day one is not a finding. It just means nobody has run it yet. Set `manual: true` for those tasks. A manual task with no log at all reports `MANUAL_OK`, not `NEVER_RAN`. After that first run, the same staleness window applies as any other task. `manual` only changes what "no log yet" means, not what "an old log" means. It does not excuse logs the checker cannot read either. A manual task whose log names were all skipped reports `LOG_UNREADABLE`, because the task has run and `MANUAL_OK` would never go stale. ## Checking the output instead of the log A log line only exists if the job ran through whatever wrapper writes the log. Run the same job another way and the log stays where it was while the real work happens. This is not hypothetical: the lane this tool came from reported CRITICAL for ten days because the task had moved to a different invocation path that wrote no log, while the thing it produces was being updated the whole time. A permanent false-stale trains you to ignore the switch, which is the one outcome worse than not having it. Point `artefact` at what the job actually produces and that signal counts too. ```json { "name": "weekly-audit", "max_age_hours": 192, "sentinel": "WEEKLY_AUDIT_OK", "artefact": { "path": "state/findings.jsonl", "format": "jsonl", "timestamp_field": "ts", "match": { "source": "^weekly-audit-\\d{4}" } } } ``` `format: "mtime"` (the default) takes the file's modification time, which is enough for a report, an export, a rendered page. `format: "jsonl"` reads a newline-delimited JSON file and takes the newest `timestamp_field` value across its records. `match` is what keeps `jsonl` honest. Each key names a field and each value is a regex the field must match, so only records the tracked job writes are counted. Without it any append to a shared file reports the lane fresh, and a false `FRESH` on a dead-man's switch is worse than the stale it replaces. The artefact is compared with when the last logged run ended, which the checker takes from the log file's last write, not with the start time in the log's name. It decides the state only when it is more than ten minutes newer than that end. The older log's sentinel is then not scored, because the artefact already shows the job produced its output. Inside the ten minutes a record is read as that same run's own output, so it cannot hide the run's `FAILED` or `HUNG` result. Where `max_runtime_hours` is set, a log with no success or failure line counts as still running until that allowance has passed. When the log is the newer signal, nothing changes. A record or modification time more than five minutes ahead of the clock is ignored as clock skew, and the detail line says how many were ignored. A relative `path` resolves against the config file's directory, and an artefact file that does not exist yet is simply no signal rather than an error. ## Jobs that start and never finish A success sentinel tells you a run finished. It cannot tell you whether a run that produced no sentinel never started, or started and wedged halfway. Both report `NO_SENTINEL`, and those are different problems with different fixes: a scheduler that skipped the job, versus a job that hangs on something. Have the job write a `start_sentinel` line as its first act, and the two split apart. A log carrying the start string but neither a success nor a failure string reports `HUNG` instead, and the detail line says how long ago it began. ```json { "name": "nightly-report", "max_age_hours": 30, "sentinel": "NIGHTLY_REPORT_OK", "start_sentinel": "NIGHTLY_REPORT_START", "max_runtime_hours": 2 } ``` `max_runtime_hours` is the allowance a run gets before an unfinished start counts as hung. Inside it the task reports `RUNNING`, which passes, so checking while a long job is genuinely mid-flight doesn't raise a false alarm. Leave it out and any unfinished start is `HUNG` immediately, which is fine when nothing you track runs long enough to overlap a check. Staleness still wins. A started-but-unfinished log old enough to breach `max_age_hours` reports `STALE`, because at that point the age is the more urgent fact. ## What "fresh" actually means `check` reports one of these states per task. `FRESH`, `STALE`, `FAILED`, `HUNG`, `RUNNING`, `NO_SENTINEL`, `NEVER_RAN`, `MANUAL_OK`, or `LOG_UNREADABLE` if the newest log file cannot be opened or none of the task's log names gives a usable run time. `FRESH`, `RUNNING` and `MANUAL_OK` pass; everything else is a finding. Staleness is checked first. A log old enough to breach its window is `STALE` regardless of what it contains. Inside the window, being recent is not automatically a pass. A log without the success sentinel still fails the check, whether that is because the job crashed and left a `failure_sentinel` behind, or because it exited without writing anything conclusive at all. ## What counts as writing the sentinel The sentinel has to open a line. A log that only mentions the string somewhere in a sentence does not pass, because that mention is exactly what a run quoting its own last failure looks like, or a summary line naming the sentinel it went looking for. Treating a mention as a success is a false `FRESH` on a dead-man's switch, which is a worse failure than the false finding it would avoid. Leading whitespace, backticks, asterisks and underscores in front of the sentinel are fine, so a log line written as `` `MY_JOB_OK` `` or `**MY_JOB_OK**` still counts. A leading byte order mark on the file does not hide the first line. A sentinel of `MY_JOB_OK` does not match `MY_JOB_OK_PENDING`. Some loggers put a timestamp at the start of every line, so the sentinel never opens one. Set `line_stamp: true` on that task and one ISO 8601 time may come first, as in `2026-01-15T09:30:00Z MY_JOB_OK` or `2026-01-15 09:30:00,123 MY_JOB_OK`. The date and time may be joined by `T` or a space, and fractional seconds and a UTC offset are accepted. The time has to open the line and the sentinel has to follow it directly, so a stamped line that only mentions the string still does not pass. `failure_sentinel` and `start_sentinel` get the same allowance. `failure_patterns` are your own regular expressions, so start one with `.*` to reach past the stamp. ## Failures the job doesn't announce `failure_sentinel` only catches a failure the job reports in a string you picked. Plenty of failures arrive in someone else's words, such as a client library printing `API Error: 401` after a token expired, or an uncaught Python exception. `failure_patterns` lists regular expressions for those lines. ```json { "name": "nightly-report", "max_age_hours": 30, "sentinel": "NIGHTLY_REPORT_OK", "failure_patterns": ["Traceback \\(most recent call last\\)", ".*API Error: 401"] } ``` Each pattern is tried with Python's `re.match` against every line of the log, so it is anchored to the start of a line but not to its end. Start a pattern with `.*` to match text anywhere in a line. A matching line marks the run `FAILED` the same way `failure_sentinel` does, and a log that also carries the success sentinel still passes. Every pattern has to compile when the config loads, so a broken regex is a config error rather than a check that never fires. ## CLI ``` python deadmans.py check [--config PATH] [--json] [--task NAME] python deadmans.py init [--config PATH] python deadmans.py selftest ``` `check` is the one to run on a schedule. It exits 0 when every task is fresh and 1 when at least one is not. A config problem, such as a missing file or an unknown `--task` name, exits 2 instead. `init` writes a starter config so there is something to edit instead of a blank file. `selftest` runs the tool's own state machine against a throwaway temp directory, a fast way to confirm the checker behaves correctly on whatever machine it was just copied to, independent of the repo's own test suite. ## Origin This is pattern 3, ["Make silent failure loud (the dead-man's switch)"](https://github.com/jimy-r/agent-workspace-architecture/blob/main/PATTERNS.md?utm_source=github&utm_medium=repo&utm_campaign=dead-mans-switch#3-make-silent-failure-loud-the-dead-mans-switch), from the [agent-workspace-architecture](https://github.com/jimy-r/agent-workspace-architecture?utm_source=github&utm_medium=repo&utm_campaign=dead-mans-switch) reference, extracted into a standalone tool. It's one of six repos published from that same workspace; the [interactive tour](https://jimy-r.github.io/agent-workspace-architecture/?utm_source=github&utm_medium=repo&utm_campaign=dead-mans-switch) walks the rest. ## Contributing Issues and pull requests are welcome. [`CONTRIBUTING.md`](CONTRIBUTING.md) has the setup, the test gate, and the one rule that is not negotiable. ## License MIT. See [LICENSE](LICENSE).