]> ###2026.09.15.1 - The cron entry now goes where Unraid actually looks for it. 2026.09.15 removed the stray user column but kept writing /etc/cron.d/failover-guard directly, and that is not a Debian-style drop-in directory here: it is dcron own spool, the one "crontab -c /etc/cron.d" owns. update_cron never even reads it. What Unraid plugins do -- dynamix and the rest, right there on any box -- is drop a .cron file in their own directory under /boot/config/plugins/ and call update_cron, which concatenates them into root crontab. So the supervisor STILL had not run after the previous release; the syslog said so five minutes later, which is the only thing that ever says so. - Installing also removes the old /etc/cron.d/failover-guard, so anyone updating from a broken version does not keep firing the bad entry next to the good one. ###2026.09.15 - The supervisor had never run. Once. The plugin wrote its /etc/cron.d/ entry with a user column, the way Debian-style crontabs take one, but Unraid runs dcron and there the sixth field is already the command. So every five minutes cron tried to execute a command called "root", failed with 127, and the thing whose whole job is to bring the watcher back after a crash, an array restart or a plugin update was never alive to do it. The watcher itself was fine -- but it survived on luck, not on care, and nothing would have said otherwise. - If you are already running this plugin, check your syslog for "exit status 127" before and after updating: that, and not the contents of the file, is what proves it is fixed. Run by hand the script exits 0 either way, so reproducing the fault outside cron is impossible. ###2026.08.21.1 - The standby copy is watched too. Until now the only thing ever checked about plan B was that a DNS record resolved, so a replica could sit for days answering errors and nothing noticed until the day it had to take over — which is exactly what happened here. Set TARGET_HEALTH_URL to a URL where the copy on this box answers (the backend directly: at rest the proxy host is disabled, so the site is not reachable locally by its public name) and it is asked every TARGET_CHECK_INTERVAL. Its state shows next to the origin on the page, and only changes are notified. - A broken standby warns, it never vetoes. Refusing to switch would turn a degraded site into an offline one, and this probe can be wrong too; the failover goes ahead with a loud notification instead. - A failover target whose address does not match this box's public IP no longer blocks the switch on its own when the standby answers. On a box behind two internet lines there is no single "our public IP", and the line traffic leaves by is not necessarily the one the record points at. ###2026.08.21 - The settings page never saved anything. The form posted to /update.htm, which is only the progress page; the file is written by /update.php. Every field looked like it was applied and nothing reached the config file. - The watcher read the configuration once at start and never again, so a change made on the page did nothing until someone restarted it. It is now reloaded whenever the file changes, and the reload is logged. - A broken resolver was reported as a dead origin. Every probe reaches the origin by name, but the "do we have internet" check only opened a socket to an IP, so with DNS down all of them failed and the origin was declared dead. That check now also requires a name to resolve, and returns "inconclusive" instead. - Failures are ignored for the first few minutes after this box boots (BOOT_GRACE, 300s). Right after a reboot the network stack and Docker are still coming up and anything measured says more about us than about the origin: a real reboot produced four failover attempts in six minutes. ###2026.08.20.8 - The watcher could die together with the shell that started it: setsid only creates a new session when the caller is not already a process group leader, so starting it by hand from a terminal left it attached to that terminal. Started from cron it survived, which is why it went unnoticed. It now always forks. ###2026.08.20.7 - A hook that refuses to run (because the site is busy and copying would capture a half-written state) left the copy pending, which meant it was retried on every pass of the watcher — a connection to the origin every few seconds for as long as the site stayed busy. There is now a configurable wait between retries, 60 seconds by default. ###2026.08.20.6 - The per-field help is back to Unraid's own mechanism: plain blockquotes that the built-in Help button in the top bar unfolds, instead of the custom question mark widget added in .5, which was not how Unraid does this. ###2026.08.20.5 - The origin health is now the first and biggest thing on the page, on the same row as the rest of the state instead of dropped onto its own line below it. - Every field has a "?" next to its label that unfolds a plain-language explanation of what it does and what happens if you get it wrong. The previous notes only showed up if you found the global help toggle, and several fields had no explanation at all. ###2026.08.20.4 - The settings page now shows what the watcher is seeing: origin healthy, origin down, or inconclusive because this server itself has no internet, plus the consecutive failure count and how many seconds ago it was checked. Until now that only appeared if you pressed the check button. The value is published by the watcher rather than probed on page load, so the page is instant and always agrees with what the watcher is acting on, and it is shown as "no recent reading" if the watcher stopped updating it. ###2026.08.20.3 - The "enabled" line in the checks output showed the raw yes/no from the config file instead of the translated word. ###2026.08.20.2 - The settings page, the checks output and the notifications now follow the language selected in Unraid (Settings -> Display Settings), instead of showing both languages at once. Spanish when the locale starts with "es", English otherwise, so any other language still gets a readable page. The log file stays in English on purpose: it is what gets pasted into a bug report. ###2026.08.20.1 - The settings page now shows the log: last 100 to 2000 lines, with refresh, clear, and an optional 5 second auto-refresh that keeps the newest lines in view. Until now the page only told you where the file was. ###2026.08.20 - First release. Watches a remote origin from your Unraid box and fails over automatically when it dies: switches the Cloudflare DNS records of a zone to a failover target and enables the matching Nginx Proxy Manager host, then returns everything when the origin comes back. - Detection in about interval x threshold seconds (10s x 3 by default), not the several minutes a cron job would take. - Three guards against false positives: a failure is not counted when this server itself has no internet, an open port alone is not accepted as healthy when a health path is configured, and a configurable number of consecutive failures is required before anything is switched. - Refuses to switch when the failover target is not reachable or its dynamic DNS has not caught up with the current public IP, since sending users nowhere is worse than leaving the outage alone. - Your data is moved by your own hook scripts, never by the plugin. The pre-failback hook runs with the local host already disabled, so nothing can be written while the copy is in flight, and a failing hook aborts the failback instead of losing whatever arrived during the outage. here HOOK_PRE_FAILBACK="" # runs before returning: here -> origin REPLICATE_INTERVAL=3600 RETRY_INTERVAL=60 # wait this long before retrying a hook that refused BOOT_GRACE=300 # ignore failures for this long after this box booted NOTIFY="yes" [ -f "$CONFIG" ] && . "$CONFIG" # The watcher is a long-lived process: without this it would keep serving the # settings it was started with, and a change made on the page would silently do # nothing until someone happened to restart it. update.php writes the file # atomically, so sourcing it mid-flight cannot read half a file. CONFIG_STAMP=$(stat -c %Y "$CONFIG" 2>/dev/null) reload_config_if_changed() { local stamp stamp=$(stat -c %Y "$CONFIG" 2>/dev/null) [ -z "$stamp" ] && return 1 [ "$stamp" = "$CONFIG_STAMP" ] && return 1 CONFIG_STAMP="$stamp" . "$CONFIG" log "Configuration reloaded" return 0 } MODE="${1:-status}" # ---- Helpers ------------------------------------------------------------ log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $*" >> "$LOG"; } # Follows the language picked in Unraid, for the things a person reads: the # checks output and the notifications. The log itself stays in English on # purpose — it is what gets pasted into a bug report. DYN_LOCALE=$(grep -oP '(?<=^locale=")[^"]*' /boot/config/plugins/dynamix/dynamix.cfg 2>/dev/null) case "$DYN_LOCALE" in es*) LANG_ES=1 ;; *) LANG_ES=0 ;; esac tt() { [ "$LANG_ES" = "1" ] && echo "$2" || echo "$1"; } notify() { [ "$NOTIFY" = "yes" ] || return 0 local subject="$1" text="$2" icon="${3:-warning}" [ -x /usr/local/emhttp/webGui/scripts/notify ] && \ /usr/local/emhttp/webGui/scripts/notify -e "Failover Guard" \ -s "$subject" -d "$text" -i "$icon" >/dev/null 2>&1 } cf_api() { local method="$1" path="$2" data="$3" if [ -n "$data" ]; then curl -s -X "$method" -H "Authorization: Bearer $CF_TOKEN" \ -H "Content-Type: application/json" --data "$data" \ "https://api.cloudflare.com/client/v4$path" else curl -s -X "$method" -H "Authorization: Bearer $CF_TOKEN" \ "https://api.cloudflare.com/client/v4$path" fi } cf_zone() { cf_api GET "/zones?name=$DOMAIN" | jq -r '.result[0].id'; } # ---- Health checks ------------------------------------------------------ # Does this box have internet at all? If not, the origin may be perfectly fine # and we simply cannot see it — and failing over would be pointless anyway, # since nobody could reach us either. # # Reaching an IP is not enough: every probe below reaches the origin BY NAME, so # a resolver that is not up yet makes them all fail and the origin gets declared # dead while it is perfectly healthy. Well known names are used on purpose, never # the origin's own: if the origin's record really disappeared that is a genuine # outage, not an excuse to look the other way. local_net_ok() { local d reachable=0 for d in 1.1.1.1 8.8.8.8; do timeout 3 bash -c "echo > /dev/tcp/$d/443" 2>/dev/null && { reachable=1; break; } done [ "$reachable" = "1" ] || return 1 for d in one.one.one.one dns.google; do [ -n "$(timeout 5 getent hosts "$d" 2>/dev/null | head -1)" ] && return 0 done return 1 } # Right after a reboot the network stack, the resolver and Docker are still # coming up, so anything we measure says more about us than about the origin. just_booted() { local up [ "${BOOT_GRACE:-0}" -gt 0 ] 2>/dev/null || return 1 up=$(cut -d. -f1 /proc/uptime 2>/dev/null) [ -n "$up" ] && [ "$up" -lt "$BOOT_GRACE" ] } origin_ports_open() { local p for p in $ORIGIN_PORTS; do timeout 6 bash -c "echo > /dev/tcp/$ORIGIN_HOST/$p" 2>/dev/null && return 0 done return 1 } # An open port only proves the machine booted. If HEALTH_PATH is set we ask the # application itself, hitting the origin by IP so we bypass any CDN in front — # otherwise the CDN would answer happily with the origin long dead. origin_app_ok() { [ -z "$HEALTH_PATH" ] && return 0 local ip body ip=$(getent hosts "$ORIGIN_HOST" | awk '{print $1}' | head -1) [ -z "$ip" ] && return 1 body=$(timeout 8 curl -s --max-time 8 --resolve "$DOMAIN:443:$ip" \ "https://$DOMAIN$HEALTH_PATH" 2>/dev/null) [ -z "$HEALTH_EXPECT" ] && { [ -n "$body" ] && return 0 || return 1; } case "$body" in *"$HEALTH_EXPECT"*) return 0 ;; *) return 1 ;; esac } # 0 = healthy · 1 = down · 2 = inconclusive (our own network is broken) origin_health() { local_net_ok || return 2 origin_ports_open || return 1 origin_app_ok || return 1 return 0 } # Is the standby copy on this box actually able to serve? Everything else here # watches the origin; without this the only thing ever checked about plan B is # that a DNS record resolves. A replica can sit there for days answering errors # and nothing notices until the day it has to take over. # # It has to be asked at a URL that works while the copy is at rest: if the proxy # host is disabled (which is the correct resting state) the site is not reachable # by its public name from here, so this points straight at the backend. # 0 = healthy · 1 = not healthy · 2 = not configured target_app_ok() { [ -z "$TARGET_HEALTH_URL" ] && return 2 local body body=$(timeout 10 curl -s --max-time 10 "$TARGET_HEALTH_URL" 2>/dev/null) || return 1 [ -z "$TARGET_EXPECT" ] && { [ -n "$body" ] && return 0 || return 1; } case "$body" in *"$TARGET_EXPECT"*) return 0 ;; *) return 1 ;; esac } # Never send traffic to a failover target that is not ready: if your dynamic DNS # has not caught up, switching sends users nowhere. Worse than staying put. target_ready() { [ -z "$FAILOVER_TARGET" ] && return 1 case "$FAILOVER_TARGET" in *[a-zA-Z]*) local ip_t ip_w ip_t=$(getent hosts "$FAILOVER_TARGET" | awk '{print $1}' | head -1) [ -z "$ip_t" ] && return 1 ip_w=$(timeout 8 curl -s https://api.ipify.org 2>/dev/null) [ -z "$ip_w" ] && return 0 [ "$ip_t" = "$ip_w" ] && return 0 # A mismatch used to be an outright veto. It cannot be, on a box # behind more than one internet line: there is no single "our public # IP" then, and the line traffic happens to leave by is not # necessarily the one the record points at. So when the standby copy # itself answers, that is better evidence than the address # comparison and it wins. if target_app_ok; then log "Failover target $FAILOVER_TARGET resolves to $ip_t and traffic leaves through $ip_w, but the standby copy answers: switching anyway" return 0 fi log "Failover target $FAILOVER_TARGET resolves to $ip_t but our public IP is $ip_w" return 1 ;; *) return 0 ;; esac } # ---- Nginx Proxy Manager ------------------------------------------------ # # Done the same way NPM does it internally: a disabled host keeps its .conf # with only the comment header. No API credentials needed. npm_conf() { echo "$NPM_DATA/nginx/proxy_host/${NPM_HOST_ID}.conf"; } npm_db() { echo "$NPM_DATA/database.sqlite"; } npm_configured() { [ -n "$NPM_DATA" ] && [ -n "$NPM_HOST_ID" ] && [ -f "$(npm_db)" ]; } npm_state() { npm_configured || { echo "n/a"; return; } sqlite3 "$(npm_db)" "SELECT enabled FROM proxy_host WHERE id=$NPM_HOST_ID;" 2>/dev/null } npm_enable() { npm_configured || return 0 [ "$(npm_state)" = "1" ] && return 0 if [ ! -s "$STATE/npm_host.conf" ]; then log "No saved copy of the NPM host config; cannot enable it" return 1 fi cp "$STATE/npm_host.conf" "$(npm_conf)" sqlite3 "$(npm_db)" "UPDATE proxy_host SET enabled=1 WHERE id=$NPM_HOST_ID;" docker exec "$NPM_CONTAINER" nginx -s reload >/dev/null 2>&1 log "NPM host $NPM_HOST_ID enabled" } npm_disable() { npm_configured || return 0 # Save the good config first: it is the only thing that lets us turn it back on. [ -s "$(npm_conf)" ] && [ "$(wc -l < "$(npm_conf)")" -gt 4 ] && cp "$(npm_conf)" "$STATE/npm_host.conf" [ "$(npm_state)" = "0" ] && return 0 printf '# ------------------------------------------------------------\n# %s\n# ------------------------------------------------------------\n' "$DOMAIN" > "$(npm_conf)" sqlite3 "$(npm_db)" "UPDATE proxy_host SET enabled=0 WHERE id=$NPM_HOST_ID;" docker exec "$NPM_CONTAINER" nginx -s reload >/dev/null 2>&1 log "NPM host $NPM_HOST_ID disabled" } # ---- DNS ---------------------------------------------------------------- # Points every A/CNAME record of the zone at the given destination, keeping the # proxied flag each record already had. dns_point_to() { local dest="$1" type zone records id name proxied res ok=0 bad=0 case "$dest" in *[a-zA-Z]*) type="CNAME" ;; *) type="A" ;; esac zone=$(cf_zone) if [ -z "$zone" ] || [ "$zone" = "null" ]; then log "Cloudflare: could not read zone for $DOMAIN" return 1 fi records=$(cf_api GET "/zones/$zone/dns_records" \ | jq -r '.result[] | select(.type=="A" or .type=="CNAME") | "\(.id)|\(.name)|\(.proxied)"') while IFS='|' read -r id name proxied; do [ -z "$id" ] && continue res=$(cf_api PUT "/zones/$zone/dns_records/$id" \ "{\"type\":\"$type\",\"name\":\"$name\",\"content\":\"$dest\",\"proxied\":$proxied,\"ttl\":1}" \ | jq -r .success) if [ "$res" = "true" ]; then log "DNS: $name -> $type $dest"; ok=$((ok+1)) else log "DNS: failed to update $name"; bad=$((bad+1)); fi done <<< "$records" [ $ok -gt 0 ] && [ $bad -eq 0 ] } origin_ip() { getent hosts "$ORIGIN_HOST" | awk '{print $1}' | head -1; } # ---- Actions ------------------------------------------------------------ run_hook() { local hook="$1" label="$2" [ -z "$hook" ] && return 0 [ -f "$hook" ] || { log "Hook $label not found: $hook"; return 1; } log "Running $label hook: $hook" if bash "$hook" >> "$LOG" 2>&1; then log "$label hook finished OK"; return 0 fi log "$label hook FAILED"; return 1 } do_failover() { [ -f "$STATE/switched" ] && return 0 log "FAILOVER: origin is down, serving from this box" if ! target_ready; then log "Failover target is not ready; refusing to switch" notify "$(tt 'Failover NOT performed' 'Failover NO ejecutado')" "$(tt 'The failover target is not reachable or your dynamic DNS is stale.' 'El destino del failover no es alcanzable o tu DNS dinámico está desactualizado.')" "alert" return 1 fi # A standby that answers wrongly is a warning, never a veto. Refusing here # would turn "the site is degraded" into "the site is down", and this probe # can be wrong too. The decision belongs to whoever reads the alert. if ! target_app_ok && [ -n "$TARGET_HEALTH_URL" ]; then log "WARNING: the standby copy is not answering as expected; switching anyway, but check it now" notify "$(tt 'Failover onto a copy that looks broken' 'Failover a una copia que parece rota')" \ "$(tt "Switching because the origin is down, but $TARGET_HEALTH_URL is not answering as expected. Check the site." "Se conmuta porque el origen está caído, pero $TARGET_HEALTH_URL no responde como se espera. Revisa el sitio.")" "alert" fi # Enable locally first: switching DNS onto a box that does not serve the # domain yet would hand users an error. npm_enable || { notify "$(tt 'Failover failed' 'Failover fallido')" "$(tt 'Could not enable the NPM host.' 'No se pudo habilitar el host de NPM.')" "alert"; return 1; } if dns_point_to "$FAILOVER_TARGET"; then date +%s > "$STATE/switched" notify "$(tt 'Failover performed' 'Failover ejecutado')" "$(tt "$DOMAIN now points to $FAILOVER_TARGET and is served from this server." "$DOMAIN ahora apunta a $FAILOVER_TARGET y se sirve desde este servidor.")" "alert" return 0 fi npm_disable notify "$(tt 'Failover failed' 'Failover fallido')" "$(tt 'DNS could not be updated; the local host was left disabled so two copies never serve at once.' 'No se pudo cambiar el DNS; el host local se dejó deshabilitado para que nunca sirvan dos copias a la vez.')" "alert" return 1 } do_failback() { [ -f "$STATE/switched" ] || return 0 local ip ip=$(origin_ip) [ -z "$ip" ] && { log "Cannot resolve the origin; failback aborted"; return 1; } log "FAILBACK: origin is back. Freezing this box before moving data" # Freeze first: from here on nobody writes on this side, so whatever the # hook uploads is the complete, final picture. npm_disable if ! run_hook "$HOOK_PRE_FAILBACK" "pre-failback"; then log "Data hook failed; keeping this box in service" npm_enable notify "$(tt 'Failback aborted' 'Failback abortado')" "$(tt 'The pre-failback hook failed, so we keep serving locally rather than lose data.' 'El enganche previo al failback falló, así que se sigue sirviendo en local en vez de perder datos.')" "alert" return 1 fi if dns_point_to "$ip"; then rm -f "$STATE/switched" date +%s > "$STATE/last_replicate" notify "$(tt 'Failback complete' 'Failback completado')" "$(tt "$DOMAIN is served from the origin again ($ip). The local host is disabled." "$DOMAIN vuelve a servirse desde el origen ($ip). El host local queda deshabilitado.")" "normal" return 0 fi log "DNS could not be updated; keeping this box in service" npm_enable return 1 } maybe_replicate() { [ -z "$HOOK_REPLICATE" ] && return 0 local now prev att now=$(date +%s) # Time for the next scheduled copy? if [ -f "$STATE/last_replicate" ]; then prev=$(cat "$STATE/last_replicate") [ $((now - prev)) -lt "$REPLICATE_INTERVAL" ] && return 0 fi # A hook is allowed to refuse: the usual reason is that the site is busy right # now and copying would capture a half-written state. When it refuses, the # success mark is deliberately not updated, so the copy stays due — but # without this backoff "still due" would mean retrying on every single pass of # the watcher, which is every few seconds, hammering the origin for as long as # the site stays busy. if [ -f "$STATE/last_attempt" ]; then att=$(cat "$STATE/last_attempt") [ $((now - att)) -lt "$RETRY_INTERVAL" ] && return 0 fi echo "$now" > "$STATE/last_attempt" if run_hook "$HOOK_REPLICATE" "replicate"; then echo "$now" > "$STATE/last_replicate" fi } # ---- Watch loop --------------------------------------------------------- # The settings page reads this instead of running its own check: it shows what # the watcher actually sees, and costs nothing to display. publish_health() { printf 'health=%s\nfails=%s\nchecked=%s\ntarget=%s\n' \ "$1" "$2" "$(date +%s)" "$(cat "$STATE/target_health" 2>/dev/null || echo off)" \ > "$STATE/health" } # Checked on its own slower clock, not on every pass: the standby breaking is a # slow kind of accident (a bad sync, a permissions change), and there is no point # knocking on it every few seconds. Only transitions are announced, so a copy # that stays broken does not turn into a notification every five minutes. check_target() { local now state prev now=$(date +%s) [ -z "$TARGET_HEALTH_URL" ] && { echo off > "$STATE/target_health"; return 0; } [ $((now - ${last_target_check:-0})) -lt "$TARGET_CHECK_INTERVAL" ] && return 0 last_target_check="$now" target_app_ok && state=ok || state=fail prev=$(cat "$STATE/target_health" 2>/dev/null) echo "$state" > "$STATE/target_health" [ "$state" = "$prev" ] && return 0 if [ "$state" = "fail" ]; then log "STANDBY COPY UNHEALTHY: $TARGET_HEALTH_URL is not answering as expected" notify "$(tt 'Standby copy is broken' 'La copia en espera está rota')" \ "$(tt "The copy on this box answered wrongly at $TARGET_HEALTH_URL. A failover right now would serve a broken site." "La copia de este servidor respondió mal en $TARGET_HEALTH_URL. Un failover ahora serviría un sitio roto.")" "alert" elif [ -n "$prev" ] && [ "$prev" != "off" ]; then log "Standby copy is healthy again" notify "$(tt 'Standby copy is healthy again' 'La copia en espera vuelve a estar sana')" \ "$(tt "$TARGET_HEALTH_URL answers as expected again." "$TARGET_HEALTH_URL vuelve a responder como se espera.")" "normal" fi } watch_loop() { local fails=0 st booting=0 last_target_check=0 log "Watcher started (every ${CHECK_INTERVAL}s, $FAIL_THRESHOLD failures to switch)" echo $$ > "$STATE/watcher.pid" trap 'rm -f "$STATE/watcher.pid"; exit 0' TERM INT while true; do reload_config_if_changed if [ "$ENABLED" != "yes" ]; then sleep "$CHECK_INTERVAL"; continue; fi if just_booted; then [ "$booting" != "1" ] && { log "This box booted less than ${BOOT_GRACE}s ago: not judging the origin yet"; booting=1; } publish_health 2 0; fails=0 sleep "$CHECK_INTERVAL"; continue fi booting=0 # Watched even while switched over: after a failover the standby IS the # live site, and it is exactly then that knowing it works matters most. # Kept after the boot grace on purpose — a copy whose container is still # starting is not a broken copy. check_target origin_health; st=$? publish_health "$st" "$fails" if [ -f "$STATE/switched" ]; then # Already switched: what we are waiting for now is the origin to # come back, and then the failback. if [ $st -eq 0 ]; then fails=$((fails+1)) if [ $fails -ge $FAIL_THRESHOLD ]; then do_failback; fails=0; sleep 60; fi else fails=0 fi sleep "$CHECK_INTERVAL"; continue fi case $st in 0) [ $fails -gt 0 ] && log "Origin responding again (discarding $fails failures)" fails=0 maybe_replicate ;; 2) [ $fails -gt 0 ] && log "No working internet or DNS on this box: not counting the failure" fails=0 ;; 1) fails=$((fails+1)) log "Origin not responding (failure $fails of $FAIL_THRESHOLD)" if [ $fails -ge $FAIL_THRESHOLD ]; then if [ -n "$HOOK_REPLICATE" ] && [ ! -f "$STATE/last_replicate" ]; then log "No replica has ever run: refusing to switch to an empty copy" else do_failover fi fails=0; sleep 60 fi ;; esac sleep "$CHECK_INTERVAL" done } watcher_running() { local pid pid=$(cat "$STATE/watcher.pid" 2>/dev/null) [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null } # ---- Entry point -------------------------------------------------------- case "$MODE" in watch) watch_loop ;; failover) do_failover ;; failback) do_failback ;; replicate) run_hook "$HOOK_REPLICATE" "replicate" && date +%s > "$STATE/last_replicate" ;; test) origin_health; h=$? w=22 printf "%-${w}s %s\n" "$(tt 'enabled:' 'activado:')" \ "$([ "$ENABLED" = "yes" ] && tt yes sí || tt no no)" printf "%-${w}s %s\n" "$(tt 'local network and DNS:' 'red local y DNS:')" \ "$(local_net_ok && echo OK || echo "$(tt FAIL FALLO)")" printf "%-${w}s %s\n" "$(tt 'origin ports:' 'puertos del origen:')" \ "$(origin_ports_open && echo OK || echo "$(tt FAIL FALLO)")" printf "%-${w}s %s\n" "$(tt 'origin app:' 'aplicación del origen:')" \ "$([ -z "$HEALTH_PATH" ] && tt 'not configured' 'sin configurar' || (origin_app_ok && echo OK || echo "$(tt FAIL FALLO)"))" printf "%-${w}s %s\n" "$(tt 'origin health:' 'salud del origen:')" \ "$([ $h -eq 0 ] && tt HEALTHY SANO || ([ $h -eq 2 ] && tt INCONCLUSIVE 'NO CONCLUYENTE' || tt DOWN CAÍDO))" printf "%-${w}s %s\n" "$(tt 'target ready:' 'destino listo:')" \ "$(target_ready && echo OK || echo "$(tt FAIL FALLO)")" target_app_ok; ta=$? printf "%-${w}s %s\n" "$(tt 'standby copy:' 'copia en espera:')" \ "$([ $ta -eq 2 ] && tt 'not configured' 'sin configurar' || { [ $ta -eq 0 ] && tt HEALTHY SANA || tt BROKEN ROTA; })" printf "%-${w}s %s\n" "$(tt 'npm host:' 'host de NPM:')" \ "$(s=$(npm_state); [ "$s" = "1" ] && tt 'enabled' 'habilitado' || { [ "$s" = "0" ] && tt 'disabled' 'deshabilitado' || echo "$s"; })" printf "%-${w}s %s\n" "$(tt 'switched:' 'conmutado:')" \ "$([ -f "$STATE/switched" ] && tt yes sí || tt no no)" printf "%-${w}s %s\n" "$(tt 'watcher:' 'vigilante:')" \ "$(watcher_running && tt running 'en marcha' || tt stopped parado)" ;; status) health=""; fails=""; checked=""; target="" [ -f "$STATE/health" ] && . "$STATE/health" jq -n --arg enabled "$ENABLED" \ --arg target "${target:-off}" \ --arg watcher "$(watcher_running && echo running || echo stopped)" \ --arg switched "$([ -f "$STATE/switched" ] && echo yes || echo no)" \ --arg npm "$(npm_state)" \ --arg last "$(cat "$STATE/last_replicate" 2>/dev/null || echo 0)" \ --arg health "${health:-}" --arg fails "${fails:-0}" --arg checked "${checked:-0}" \ '{enabled:$enabled,watcher:$watcher,switched:$switched,npm_host:$npm, last_replicate:$last,health:$health,fails:$fails,checked:$checked, target:$target}' ;; *) echo "usage: $0 {watch|failover|failback|replicate|test|status}"; exit 1 ;; esac ]]> a63fa1e7e5eba930ab3dd0142e675ac8fc14554b0968a58d4ca479ca9f72f889 /dev/null) if [ "$ENABLED" != "yes" ]; then # Disabled on purpose: stop the watcher if it is still around. if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then kill "$pid" 2>/dev/null echo "$(date '+%Y-%m-%d %H:%M:%S') Watcher stopped (plugin disabled)" >> "$LOG" fi rm -f "$STATE/watcher.pid" exit 0 fi if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then exit 0; fi # --fork is not optional: without it setsid only creates a new session when the # caller is not already a process group leader, so launching this from an # interactive shell leaves the watcher in that shell's group and it dies with it. # Started from cron it happened to survive, which is exactly the kind of bug that # only shows up for whoever runs it by hand. setsid --fork bash "$BASE/scripts/failover-guard" watch >/dev/null 2>&1 < /dev/null & echo "$(date '+%Y-%m-%d %H:%M:%S') Watcher started by supervisor" >> "$LOG" ]]> 751dbea6c51e6a08669bf243f61a707cce81f725dd463df1301363e4d1c1b8e8 Display Settings), read straight from dynamix.cfg. Spanish if the locale starts with "es", English otherwise, so any other language falls back to something readable rather than to an empty page. */ $plugin = "failover-guard"; $cfgfile = "/boot/config/plugins/$plugin/config"; $script = "/usr/local/emhttp/plugins/$plugin/scripts/failover-guard"; $dyn = @parse_ini_file("/boot/config/plugins/dynamix/dynamix.cfg"); $locale = $dyn['locale'] ?? ''; $ES = (stripos($locale, 'es') === 0); /* t() picks the string for the active language. Kept deliberately simple: a full language-pack integration would mean shipping .txt files that only get loaded when the matching pack is installed, and this page has to work either way. */ function t($en, $es) { global $ES; return $ES ? $es : $en; } $cfg = file_exists($cfgfile) ? parse_ini_file($cfgfile) : []; function v($cfg, $k, $d = "") { return htmlspecialchars($cfg[$k] ?? $d); } /* Field help is written as plain markdown blockquotes below each field. Unraid turns every blockquote into

, hidden by CSS and unfolded by the built-in Help button in the top bar. Use that mechanism — do not build a custom one. */ $status = json_decode(shell_exec("bash " . escapeshellarg($script) . " status 2>/dev/null"), true) ?: []; $switched = ($status['switched'] ?? 'no') === 'yes'; $watcher = $status['watcher'] ?? 'stopped'; $lastrep = (int)($status['last_replicate'] ?? 0); /* Health as last seen by the watcher, not a fresh probe: showing it live would make this page wait several seconds every time it loads, and it would also disagree with what the watcher is actually acting on. */ $health = $status['health'] ?? ''; $fails = (int)($status['fails'] ?? 0); $checked = (int)($status['checked'] ?? 0); $age = $checked > 0 ? time() - $checked : -1; /* A reading nobody has refreshed is not a reading. If the watcher died, the last value on disk could be hours old and would be read as if it were current. */ $stale = ($age < 0 || $age > 120); switch ($health) { case '0': $hlabel = t('HEALTHY', 'SANO'); $hclass = 'fg-ok'; break; case '1': $hlabel = t('DOWN', 'CAÍDO'); $hclass = 'fg-bad'; break; case '2': $hlabel = t('INCONCLUSIVE', 'NO CONCLUYENTE'); $hclass = 'fg-warn'; break; default: $hlabel = t('unknown', 'desconocido'); $hclass = 'fg-warn'; break; } if ($stale) { $hlabel = t('no recent reading', 'sin lectura reciente'); $hclass = 'fg-warn'; } /* The standby is shown next to the origin on purpose. A failover system where nobody ever looks at plan B is how a replica ends up broken for days. */ $target = $status['target'] ?? 'off'; ?>
: 0): ?> : / : : 0): ?> :

**:** : > ### **:** : > Not your public domain. If your domain goes through Cloudflare or any CDN, that domain answers even when the server behind it is dead, so watching it would tell you nothing.', 'La máquina que se vigila: tu VPS o servidor remoto, por su propio nombre de host — por ejemplo vps.example.com. No tu dominio público. Si el dominio pasa por Cloudflare o cualquier CDN, ese dominio responde aunque el servidor de detrás esté muerto, así que vigilarlo no diría nada.') ?> **:** : > **:** : > **:** : > ### **:** : > todos los registros A y CNAME de esa zona, así que usa un dominio dedicado a este sitio, no uno compartido con otros servicios.') ?> **:** : > **:** : > API Tokens. Give it Zone:Read and DNS:Edit, and restrict it to this one zone. Do not use a Global API Key: that one can do anything to your whole account, and this plugin only needs to edit these records.', 'Se crea en Cloudflare, en Mi perfil -> Tokens de API. Dale Zone:Read y DNS:Edit, y limítalo a esta única zona. No uses la Global API Key: esa puede hacer cualquier cosa en toda tu cuenta, y este plugin solo necesita editar estos registros.') ?> **:** : > **:** : > ### **:** : > **:** : > **:** : > Leave that host disabled in normal times: the plugin enables it only during a failover, so two copies of your site never accept writes at the same time.', 'El número identificador del proxy host de NPM que sirve este sitio desde tu Unraid. Abre ese host en la interfaz de NPM y mira el número al final de la barra de direcciones. Deja ese host deshabilitado en condiciones normales: el plugin lo habilita solo durante un failover, para que nunca haya dos copias de tu sitio aceptando escrituras a la vez.') ?> ### **:** : > the copy on this box answers, checked every few minutes so you find out that plan B is broken before you need it, not during the outage. Point it straight at the backend (the container address), not at the public domain: while the proxy host is disabled — which is the correct resting state — the site is not reachable here by its public name. Leave empty to skip this check.', 'Opcional pero muy recomendable: una URL donde responda la copia de este servidor, que se comprueba cada pocos minutos para que te enteres de que el plan B está roto antes de necesitarlo, no durante la caída. Apúntala directamente al backend (la dirección del contenedor), no al dominio público: mientras el proxy host está deshabilitado — que es el reposo correcto — el sitio no es alcanzable aquí por su nombre público. Déjala vacía para no hacer esta comprobación.') ?> **:** : > **:** : > ### **:** : > **:** : > from this box back up to the origin, before handing the site back. This is the important one: while the domain pointed here, this is where new data was written, so pulling from the origin instead would erase it. It runs with the local proxy host already disabled, so nothing can be written mid-copy, and if it fails the failback is cancelled and this box keeps serving.', 'Ruta a un script tuyo que copia los datos desde este servidor de vuelta al origen, antes de devolverle el sitio. Este es el importante: mientras el dominio apuntó aquí, es aquí donde se escribieron los datos nuevos, así que traerlos del origen los borraría. Se ejecuta con el proxy local ya deshabilitado, para que nada pueda escribirse a mitad de copia, y si falla se cancela el failback y este servidor sigue sirviendo.') ?> **:** : > **:** : > **:** : > :
###
###
> : `/var/log/.log` ]]> 0c82e3d25fd60b67fc758ffa1849cccb75005ddd0a9ca912e001e9d43e67add4 &1"); break; case 'status': header('Content-Type: application/json; charset=utf-8'); echo shell_exec("bash " . escapeshellarg($script) . " status 2>&1"); break; case 'log': header('Content-Type: text/plain; charset=utf-8'); if (!file_exists($logfile)) { echo "No log yet / Todavía no hay registro"; break; } $lines = (int)($_POST['lines'] ?? 200); if ($lines < 20) $lines = 20; if ($lines > 2000) $lines = 2000; echo shell_exec("tail -n $lines " . escapeshellarg($logfile) . " 2>&1"); break; case 'clearlog': header('Content-Type: text/plain; charset=utf-8'); file_put_contents($logfile, ""); echo "Log cleared / Registro vaciado"; break; default: http_response_code(400); echo "unknown action"; } ]]> af7c9ae9479f79b391992b45029e29189d45a76e61e3fffa997e38c394de2f62 mkdir -p &cfgdir;/state # Seed a config on first install so the page has something to read. if [ ! -f &cfgdir;/config ]; then cat > &cfgdir;/config <<'CFG' ENABLED="no" ORIGIN_HOST="" ORIGIN_PORTS="443" HEALTH_PATH="" HEALTH_EXPECT="" DOMAIN="" FAILOVER_TARGET="" CF_TOKEN="" CHECK_INTERVAL="10" FAIL_THRESHOLD="3" NPM_DATA="" NPM_CONTAINER="Nginx-Proxy-Manager-Official" NPM_HOST_ID="" TARGET_HEALTH_URL="" TARGET_EXPECT="" TARGET_CHECK_INTERVAL="300" HOOK_REPLICATE="" HOOK_PRE_FAILBACK="" REPLICATE_INTERVAL="3600" RETRY_INTERVAL="60" BOOT_GRACE="300" NOTIFY="yes" CFG chmod 600 &cfgdir;/config fi # The watcher is a long-lived process; this keeps it alive across crashes, # array restarts and plugin updates. # # THE UNRAID WAY: a .cron file under the plugin's own config directory, and then # update_cron. That script concatenates /boot/config/plugins/*/*.cron into root's # crontab -- it does NOT read /etc/cron.d/, which on Unraid is dcron's spool # (crontab -c /etc/cron.d), not Debian's drop-in directory. # # Writing straight into /etc/cron.d/ is what this plugin used to do, and it never # worked: the entry also carried a user column, so every five minutes dcron tried # to run a command called "root" and exited 127. The supervisor -- whose entire # job is to bring the watcher back after a crash, an array restart or a plugin # update -- had never run once. No user column here either: these lines end up in # root's crontab, where the sixth field is already the command. cat > &cfgdir;/&name;.cron <<'CRON' # Keep the watcher alive across crashes, array restarts and plugin updates: */5 * * * * /usr/local/emhttp/plugins/failover-guard/scripts/supervise &> /dev/null CRON # Clean up after the versions that wrote it in the wrong place, or dcron would # keep firing the broken entry next to the good one. rm -f /etc/cron.d/&name; [ -x /usr/local/sbin/update_cron ] && /usr/local/sbin/update_cron &plugdir;/scripts/supervise >/dev/null 2>&1 || true echo "" echo "----------------------------------------------------" echo " failover-guard has been installed." echo " Configure it at Settings -> Failover Guard." echo " Version: &version;" echo "----------------------------------------------------" echo "" PID=$(cat &cfgdir;/state/watcher.pid 2>/dev/null) [ -n "$PID" ] && kill "$PID" 2>/dev/null rm -f &cfgdir;/&name;.cron rm -f /etc/cron.d/&name; # por si viene de una version anterior [ -x /usr/local/sbin/update_cron ] && /usr/local/sbin/update_cron rm -rf &plugdir; echo "" echo " failover-guard removed. Settings kept in &cfgdir;" echo ""