site-watchdog-sweep: Every 5min, the Consolidation Story

The rule in one sentence

One Python sweeper, site_watchdog_sweep.py, runs every 5 minutes from cron job 57264ea07715, probes 6 things in a fixed order, stays silent on green, and emits exactly one structured Telegram message on red — replacing 7 legacy per-service watchdogs that used to do the same job in eight noisy scheduler slots.

What follows: the six checks, why one job replaced eight, the “sweep-all-or-nothing” design that lets it stay silent when nothing is wrong, and why the cron output is terse on green and only shouts on red.

Why one job replaced eight

Before 2026-07-14T01:52:17Z, the operator surface was held up by 8 separate 5-minute jobs — abs-backend-watchdog (:8793), abs-site-static-watchdog (:8792), atsrd-watchdog (:18071), blog-skill-watchdog (:8791), cloudflared-watchdog (a full-agent cron), plus dashboard, mockup-studio, and an off-the-clock alert sidekick. Each had its own probe and restart script; they could not talk to each other. When the tunnel was down, three downstream checks would fire, three restart scripts would race, and three Telegram alerts would land within 30s. Real outages blended in with the cascade of red herrings.

A second bug class had no watchdog at all: the homepage rendering with browser defaults because a CSS/JS asset failed to decode (proxy bug, CF content-encoding mismatch, build truncating output — all showed up as fully-loaded HTML with Times New Roman). Per-service checkers probe /healthz; the home page is a separate surface.

Consolidation was two moves: fold the eight noisy watchdogs into one ordered sweep, and add the one missing check. The script is Python because joining cascade-probe output with service-level results into one Telegram message is the text processing bash is bad at, and the cascade probe needs Playwright — reusing /opt/data/scripts/cascade_probe.py beats inventing one inline.

The probe matrix

Each sweep pass runs the six checks in this fixed order. Order matters: get the tunnel up first, otherwise every downstream check fails and the alert is just tunnel noise.

#	Check	Surfaces probed	Health probe	Two-signal?	Cold-start wait
1	cloudflared	tunnel 800a6c54-c905-40a7-beb3-d06212b6cf96 (port :20241)	http://127.0.0.1:20241/ready == 200	yes (proc + http)	20s
2	cascade:agenticbotsitter	https://agenticbotsitter.com/ (the bug class the old watchdogs missed)	cascade_probe.probe_live_site(url, selectors=("html","body"), tokens=("--bg","--ink","--sdp-bg","--font-body")) — sheets parsed, no requestFails, body fontFamily not browser-default, all 4 design tokens resolve	n/a (read-only)	Playwright load
3	abs-site-static	Caddy/python serving /var/www/agenticbotsitter (:8792)	http://127.0.0.1:8792/.well-known/security.txt == 200 (the bash-watchdog’s exact probe)	port + http	20s
4	abs-backend	ABS FastAPI origin (:8793)	http://127.0.0.1:8793/healthz == 200	http	15s
5	atsrd	ATS RD FastAPI origin (:18071)	http://127.0.0.1:18071/ == 200	yes (proc + http, cold start with DB + templates)	25s
6	blog-skill	Flask behind bestshirtdesigns.com (:8791)	http://127.0.0.1:8791/healthz == 200	http	15s

The slowest is 25s because atsrd cold-starts DB and templates; others 15s-20s. Worst case 85s plus cascade-probe; green passes finish inside 5s because the waits only matter when a service restarts.

The “two-signal” column matters: some services can answer /healthz while being zombies. cloudflared and atsrd check both process and metrics; others use a single http probe because that’s been enough.

The sweep-all-or-nothing design

Two rules apply to every check, and they are why the operator surface can stay quiet:

1. Each restart is local to that pass. No state file, no daemon, no shared lockfile. The script can run twice in 30s without fighting itself because every restart calls subprocess.Popen(..., start_new_session=True) (the nohup+disown shape) and walks away. Idempotency is a property of the helper, not something to enforce.
2. The pass is the unit of truth. Either every check is ok (zero stdout, exit 0, cron delivers nothing) or at least one check is not (status != "ok", exit 1, one structured alert). No in-between.

The restart_attempt helper owns the “kill stuck + restart + wait” sequence in the exact same order the per-service bash watchdogs did: pgrep -f <pattern> | xargs -r kill -9, then ss -tlnp | grep ':PORT ' | grep -oP 'pid=\\K[0-9]+' | xargs -r kill -9 for port-holders, then a fresh start_new_session=True process, then a poll for /healthz returning 200 (or the configured health port listening). Restart logic is preserved bit-for-bit; only the place it runs from changed.

When the sweep finishes, the main loop does one of two things:

• All ok → return 0. Cron sees empty stdout. The cron subsystem’s deliver=local rule (paired with no_agent: true) means no message is delivered. Silent.
• Any fault → sys.stdout.write(render_alert(results, header="site-watchdog-sweep")) returns 1. The cron scheduler wraps that stdout into a single Telegram message via the local-delivery code path. One alert, ever, per pass.

What the alert actually looks like

The formatter lists the healthy checks first (so you can scan and confirm what stayed up) and drills into each fault with three lines: service (status), detail (what failed), → next (what to investigate). For the cascade probe, next points to /tmp/abs-broken-viewport.png (the Playwright screenshot), /opt/data/scripts/test_abs_css_load.py (diagnosis hints), and the Cloudflare cache purge URL for zone 3ec4041668dd5f152058d9b73af60951. The sweeper does not auto-purge CF — the alert surfaces the path and you run the command.

A real 2026-07-14T22:22 sweep from the audit log shows the green-path shape — 4s of work, two lines:

[2026-07-14T22:22:23Z] sweep started
[2026-07-14T22:22:27Z] sweep done: 0 faults

Empty stdout, exit 0, no Telegram. The cascade:agenticbotsitter failure shape — N asset(s) failed to decode, body fontFamily='Times New Roman' — is the ABS-2026-07-14-CSS bug class, caught at the same 5-minute cadence as the rest.

Terse on green, shouts on red

The terse-on-green / shout-on-red split is the no_agent rule from the cron-jobs guide in disguise: empty stdout on healthy runs, structured alert on faults. Agent-based jobs that produce output every run produce noise; jobs that never produce output make “did it run?” indistinguishable from “is everything broken?”

For this sweeper, three reasons justify the split:

1. 5-minute cadence amplifies noise. Eight old watchdogs at 5-minute cadence was ~2300 runs/day. Even a 5% false-positive rate is ~115 needless alerts. One consolidated sweeper with one alert per pass cuts that floor.
2. Empty stdout lets cron-health-monitor watch the sweep itself. The sweep’s own jobs.json record is last_run_at, last_status, repeat.completed — i.e. the scheduler is the seventh probe. The cron-health-monitor skill (3×/day) reads those fields and can flag a stuck sweep just as easily as a stuck service. That is what “the cron DB schema itself” is doing in this story: it is the meta-check.
3. One structured alert beats eight partial ones. When the tunnel is down, you do not want 5 red lines stacked; you want one message listing every check as fixed / failed / ok. That triage is the whole point of consolidation.

The cleanup tail: sweep_old_watchdogs.py

The seven legacy crons are not deleted at consolidation time. They are paused for a 7-day fallback so that if the new sweeper misbehaves you can re-enable an old watchdog and roll back. /opt/data/scripts/sweep_old_watchdogs.py knows the seven IDs by hardcoded list — never a regex, never a wild card — and only deletes a target if paused AND paused_at < (now - 7d). A backup of jobs.json is written before any change; every deletion appends to /opt/data/logs/sweep_old_watchdogs.audit.log. Run --dry-run first.

The first version of that script was buggy: it scanned jobs.json for any paused cron older than 7d and deleted it, eating 24 intentionally-paused crons unrelated to the consolidation. The rewrite only inspects jobs by id; treat the second version as the only safe one.

What NOT to do

• Do not let a watch-dog into the consolidation that depends on agent context. This sweeper is no_agent: true; the script is the job. Adding a prompt would put us back where the cloudflared watchdog was — full agent, prompt-shaped, fragile.
• Do not auto-purge CF on cascade-probe failure. The bug class has many causes; surface it for triage.
• Do not refactor the per-service restart contracts. They are preserved bit-for-bit; consolidate the caller, not the callees.
• Do not run sweep_old_watchdogs.py without --dry-run first on the day of cut-over; the early days of the fallback window are when you find out whether the new sweeper is healthy enough to keep.

Sources

• Hermes Agent cron docs
• ABS companion: Hermes Agent cron jobs: authoring, scheduling, the self-contained prompt rule
• ABS companion: ABS cron job self-contained prompts: the rule
• /opt/data/scripts/site_watchdog_sweep.py, /opt/data/scripts/cascade_probe.py, /opt/data/scripts/sweep_old_watchdogs.py
• /opt/data/cron/jobs.json — job 57264ea07715
• /opt/data/skills/devops/cron-health-monitor/SKILL.md

Last verified: 2026-07-14 against the live cron store on this VPS (job 57264ea07715, repeat.completed: 215, last_run 2026-07-14T23:22:30Z, sweep log shows 15 consecutive green passes from 21:58 to 23:22 UTC).