Cron Watchdogs vs Cron Monitors: When Each Fits and the Silent-Watchdog Trap

Two cron jobs look identical from the outside: a scheduled run, a Telegram message, a last_status: ok or error. They are not. One is doing a job; the other is checking on a job. Calling the second one a watchdog — or building a watchdog that only does the second — is the most common way a “self-healing” cron setup ends up silently broken for three days.

This guide is the distinction, the failure shape when the distinction is wrong, and the three shapes on each side where the choice is unambiguous.

The one-sentence distinction

A monitor watches a condition and alerts. A watchdog watches a condition, takes a corrective action if the condition is wrong, and alerts only if the action did not work.

The verb difference is the whole load. Monitor → alert. Watchdog → fix, then alert if fix fails. If your “watchdog” never takes a corrective action, it is a monitor. Rename the file.

Why the distinction collapses in practice

A monitor (probe + alert) and a watchdog (probe + fix + alert-if-fix-fails) look identical while everything is healthy. The differences surface only when something breaks. Two forces blur the names:

1. LLM-driven “silent on healthy” prompts are never fully silent. An agent prompt with If probe returns 200, exit with no output still produces reasoning alongside [SILENT]; the cron subsystem wraps stdout as a Cronjob Response.
2. The bash wrapper pattern works for both shapes. A no_agent script that exits 0 silent on healthy and prints ALERT: … on failure serves both. The verb at the start — alert vs fix-then-alert — is the only difference.

The bug class when the verb is wrong: 100% silent uptime, 100% broken service. Logs look fine. Telegram is quiet. The service has been down for three days.

The monitor side: 3 shapes where alert-only is right

A monitor’s job is to observe a condition you cannot or should not fix automatically and to wake a human when the condition is wrong. The condition is too expensive to act on, too risky to act on blindly, or outside the cron’s authority.

1. Cost-sensitive probes (R2 gap detection)

Daily snapshot uploads to R2 can fail for reasons you cannot fix from the cron — IPv6 ENETUNREACH, SignatureDoesNotMatch after rotation, downstream JSON parse failures. The fix lives upstream (rotate the key, fix the consumer). The right pattern is a gap-detection monitor: list the R2 prefix, parse the latest daily-YYYY-MM-DD key, alert if the latest is >36 hours old. Verified incident: 4-day gap in swag-media:Websites/AI Model Matrix/daily/ (2026-06-29 to 2026-07-02) — boto3.put_object returned success but the upload never landed. A watchdog that re-uploaded would have failed the same way and hidden the underlying problem.

2. Security and compliance scans (secrets, CVEs, HSTS preload, CSP reports)

Secret scanning, dependency CVE scanning, HSTS preload watch, CSP report digest all produce findings that need human judgment. A watchdog that auto-deletes suspected secrets or auto-bumps dependencies makes decisions a human should make. The right shape is a finding-list monitor: scan, format, alert. The operator decides.

3. Operational state outside the cron’s authority (TLS cert expiry, third-party API health)

TLS cert expiry — fix is certbot renew, not a cron’s call at 03:14 UTC. Third-party API health (Cloudflare status, R2 reachability, vendor status pages) — fix is on the vendor’s side. DNS for a hostname you don’t control — fix is at the registrar. Conditions you observe but cannot fix from the cron. Alert-only is correct.

The watchdog side: 3 shapes where only fix-then-alert is correct

A watchdog’s job is to keep a service alive without paging a human and to alert only when its own revival attempts have failed. The condition is local, the fix is local, and the fix is cheap enough to run unattended.

1. Service restart loops (cloudflared, abs-news-cron, atsrd)

A long-running daemon — cloudflared, abs_news_cron.py, the atsrd API server — can die for reasons recoverable with a systemctl restart or pkill -f + relaunch: an OOM, a network blip, a missed supervisor heartbeat. Manual restart at 03:14 UTC costs the operator sleep.

The canonical watchdog on this VPS, job 360e63a4ffc3, runs every 5 minutes. Three properties make it a watchdog, not a monitor: healthy → silent; one restart fixed it → silent; two failed restarts → alert with diagnostics. Same shape applies to a2a-relay-watchdog (cdecc9e63bae), a2a-poller-watchdog (5dd963b543de), atsrd-watchdog (386da725ab60), blog-skill-watchdog (f0690c3b821c) — seven watchdogs on this VPS, each ~30 lines of bash.

2. Zombie process reaping (stale Python/Node workers)

A long-lived worker can become a zombie — listed in ps, holding a port, but its event loop is wedged. kill -0 <pid> returns 0 because the PID exists; the worker cannot serve a request. The watchdog probes a real liveness check (GET /healthz returning 200 in <2s), then escalates: SIGTERM → wait 5s → check again → SIGKILL → relaunch → recheck. Alert only if the relaunched worker also fails the probe. The escalation ladder is the watchdog’s value-add over the monitor.

3. Disk usage cap (log rotation, temp file janitor)

Disk fills up for boring reasons: log files that should have rotated, a temp directory of orphaned artifacts. None of this needs a human, but a full disk will break other services. The same shell pattern applies: probe (df / --output=pcent), rotate (find … -delete, journalctl --vacuum-time), recheck, alert with du breakdown only if rotation didn’t free enough. Silent-healthy, silent-fixed, alert-with-breakdown-only.

The failure shape of a misclassified cron

Misclassification	What happens	Signal
”Watchdog” that only alerts	Service dies, cron fires every 5 min, Telegram lights up, but nothing is restarted. Operator paged at 03:14 UTC for an OOM a systemctl restart would have fixed.	High alert volume + service downtime + script never calls a fix primitive.
”Monitor” that tries to fix	R2 gap detected, monitor tries to re-upload. Fails for the same reason the original failed (IPv6 ENETUNREACH). Monitor hides the underlying problem.	Repeated “fixed” alerts + gap still present.
Watchdog whose fix is a silent no-op	Daemon dead, watchdog restarts, daemon dies 30s later, watchdog restarts, daemon dies again. Telegram silent because “watchdog is happy” — service at 0% uptime for six hours.	100% silent cron, 100% broken service. Check systemctl status <unit> directly.

The third row is the silent-watchdog trap: a watchdog without a failure-of-the-fix alert path is a monitor that lies. The cron-health-monitor skill (P34, P31) calls this out for the dc48d3018a78 security-profile case. A watchdog whose fix path can silently fail is worse than a monitor, because the monitor at least wakes a human.

The decision rule, in one line

If the cron can take a corrective action locally and cheaply, it is a watchdog. If the corrective action is risky, expensive, or outside the cron’s authority, it is a monitor.

• Tunnel daemon down? Watchdog (restart).
• Worker wedged? Watchdog (SIGTERM → SIGKILL → relaunch).
• Disk over 85%? Watchdog (rotate → recheck).
• R2 daily snapshot missing? Monitor (fix is upstream).
• Secret in committed code? Monitor (fix needs human judgment).
• TLS cert expiring in 14 days? Monitor (fix is certbot renew).
• CVE in a dependency? Monitor (fix is a version bump).

Verification checklist

#	Command	What it proves
1	grep -l 'systemctl restart|pkill|relaunch' /opt/data/scripts/*.sh	A watchdog has a fix primitive in its wrapper. Empty = monitor in watchdog clothing.
2	grep -c 'CODE = "200"' /opt/data/scripts/cloudflared_watchdog.sh	Two+ matches (probe + recheck) = watchdog shape.
3	hermes cron list | jq '.[] | select(.script != null) | .id' | wc -l	Total no_agent cron count. Reconcile against the 7-watchdog / 11-monitor split.
4	ls /opt/data/cron/output/<watchdog_id>/ | tail -5	Watchdog output dir. Mostly empty (silent on healthy) or short (ALERT: lines). Long verbose bodies = monitor misclassified as watchdog.
5	ls /opt/data/cron/output/<monitor_id>/ | tail -5	Monitor output dir. Structured findings bodies. Empty = probe is broken.

If #1 returns empty for a script that’s supposed to be a watchdog, rename it. If #5 returns empty for a monitor, the probe is broken — fix it before the next alert fires.

What NOT to do

• Don’t call a probe-and-alert script a “watchdog.” The name carries an expectation of corrective action. A monitor with the watchdog name will get watchdog-shaped prompts that don’t match its actual behavior.
• Don’t add fix primitives to a monitor. R2 gap detection, CVE scanning, TLS cert probes should not attempt re-upload, version bump, or certbot renew from inside the cron.
• Don’t make a watchdog fully silent. The ALERT: … after N attempts line is the only thing that distinguishes “watchdog working” from “service dead, watchdog not running.”
• Don’t auto-fix on first observation. A watchdog should probe twice before fixing (network blip, GC pause). The cloudflared wrapper restarts only after the second non-200.

Sources

• Hermes Agent cron docs
• ABS companion: ABS Cron Job Self-Contained Prompts: The Rule
• ABS companion: Hermes Agent Cron Job Management
• /opt/data/skills/devops/cron-health-monitor/SKILL.md — P34, P31, and the no_agent wrapper table.

Last verified: 2026-07-14. 7 watchdogs on this VPS (Dashboard ab4ccd928030, Cloudflared 360e63a4ffc3, blog-skill f0690c3b821c, Mockup Studio, atsrd 386da725ab60, abs-site-static 02fb666c7f40, abs-backend 3c987e8db1cf); 11 monitors (Memory Audit, Secret/CVE/HSTS/CSP/drift scans, plus sweep dispatchers).