ABS Regression Suite: Eight Failure Modes and Their Cheap-First Checks

The ABS regression suite is eight Python scripts under /opt/data/scripts/. Each one encodes a specific bug class that bit the site once; the suite grows on every verified regression. This guide is the matching troubleshooting reference: per script, here is the failure shape (verbatim from the script), the cheap-first check (the smallest thing to run), and the canonical fix.

If a script is currently passing, the section still applies the moment it goes red. Today (2026-07-14), only test_abs_analytics_backend.py is red — the watchdog cron it depends on is paused. The other seven are green.

Run the whole suite

for s in search_bar search_filter css_load backend_security cloudflare_baseline \
         daily_briefing mobile_nav analytics_backend; do
  printf '\n=== test_abs_%s.py ===\n' "$s"
  python3 "/opt/data/scripts/test_abs_${s}.py" 2>&1 | tail -5
done

A Summary: N sections pass, M sections fail and an ## ❌ FAILURES block is the one to open. Match the ✗ lines against the failure-shape text below.

How to use each section

Each script section below uses the same four blocks: failure shape, cheap-first check, canonical fix. Open the failing script’s section first; the matching ✗ line is your anchor.

Quick-reference table:

#	Script	Targets	Failure class
1	test_abs_search_bar.py	dist/ build output	Build pipeline dropped SearchBar markup
2	test_abs_search_filter.py	live (Playwright)	CSS or JS bug hides filtered list
3	test_abs_css_load.py	live (Playwright)	Content-Encoding header leak
4	test_abs_backend_security.py	live HTTPS	Tunnel route / cookies / sanitization
5	test_abs_cloudflare_baseline.py	CF API + YAMLs	Desired ↔ live tunnel drift
6	test_abs_daily_briefing.py	module + SQLite	UTC↔MDT math / empty-DB / quiet-day
7	test_abs_mobile_nav.py	live HTTPS	Page missing hamburger / drawer / cache-bust
8	test_abs_analytics_backend.py	local + cron JSON	Backend down / watchdog paused / zombie check

1. test_abs_search_bar.py — build-output search-bar contract

Failure shape: dist/<page>/index.html missing data-abs-search form or 0 inputs with id='absSearchInputNews'.

Cheap-first check: grep -E 'data-abs-search|absSearchInput|absSearchIndex' /srv/abs-site/dist/<page>/index.html — empty grep means the build dropped the markup; if source has it but dist doesn’t, the page stopped importing SearchBar.astro.

Canonical fix: npm run build from /srv/abs-site. If still missing after a green build, restore the SearchBar.astro import in the page.

2. test_abs_search_filter.py — live-list filter behavior

Failure shape: ReferenceError: debounce is not defined (Node crash), or status text says “Showing N of M” but page renders all items (CSS specificity class — [hidden] overridden by Astro-scoped display: grid).

Cheap-first check: DevTools at the failing URL: document.querySelector('.sdp-queue-item').hidden should be true and getComputedStyle(...).display must be 'none'. If hidden === true but display !== 'none', it is the CSS class.

Canonical fix: CSS: [hidden] { display: none !important; } in src/styles/global.css. ReferenceError: hoist the helper above the IIFE. > Tunable: the >= minimum-match counts in this file (e.g. mac query on /reviews/ is now >= 2 after the sample-macmini-* fixtures were deleted 2026-07-14). Retune against the new corpus; do not lower.

3. test_abs_css_load.py — CSS asset + token resolution on the live site

Failure shape: BODY BG: got rgb(255, 255, 255) (expected rgb(252, 240, 225)) or REQUIRED VAR: --sdp-bg not resolved on body. Equivalent to “ABS looks like black Times New Roman — no design system.”

Cheap-first check: curl -sSI https://agenticbotsitter.com/scripts/app.css | grep -iE 'content-encoding|content-length' — Content-Encoding: gzip on a non-gzipped body is the bug class (proxy decode leak).

Canonical fix: Strip Content-Encoding and Content-Length from the upstream response in /opt/data/abs/backend/app/proxy.py when decoded. Then CF-purge the asset.

4. test_abs_backend_security.py — security contract of /track, /api/subscribe, /admin

Failure shape: /track returned HTTP 404 (expected 200) is the CRIT — tunnel routes ABS to static origin :8792 instead of FastAPI :8793. Other shapes: utm XSS persistence, /confirm Content-Type wrong, admin cookie missing Secure.

Cheap-first check: curl -X POST https://agenticbotsitter.com/track -d '{"path":"/_diag"}' -w '\nHTTP %{http_code}\n'. 200 → bug is data/response layer. 404 → tunnel regressed.

Canonical fix: Tunnel: re-apply ingress to service: http://localhost:8793. utm: enforce ^[A-Za-z0-9_.+-]{1,100}$ at insert. HTML: html.escape() in /confirm and /unsubscribe. Cookies: ABS_SECURE_COOKIES=1 and restart.

5. test_abs_cloudflare_baseline.py — baseline YAMLs vs live tunnel state

Failure shape: :8792 ROUTING: ...agenticbotsitter.com.yaml still maps to :8792, LIVE TUNNEL: ... → service http://localhost:8792 (expected :8793), or DRIFT: desired ↔ live.

Cheap-first check: Read the YAML at /opt/data/cloudflare-sites/agenticbotsitter.com.yaml, then GET the live ingress from CF API. The diff is the answer. Test was added after a 9-day analytics blackout caused by this exact drift.

Canonical fix: Re-apply desired state via CF API PUT or hermes cf-desired-state apply. For the :8792 historical assertion: edit the YAML so ABS hostnames point at :8793, then re-apply.

6. test_abs_daily_briefing.py — daily Telegram briefing content

Failure shape: wrong MDT day window, format_message(empty) returned '' (expected 'No traffic yesterday.'), briefing crashed on empty DB, or page_views total includes affiliate_click rows (double-counted).

Cheap-first check: python3 -c "import sys; sys.path.insert(0,'/opt/data/scripts'); import abs_daily_briefing as b; s,e=b.yesterday_window_mdt(); print(b.format_message(b.aggregate(s,e)))". Window correct + message wrong → formatter; both wrong → UTC↔MDT math.

Canonical fix: Wrong window: MDT_OFFSET = timedelta(hours=-6) and datetime.min.replace(hour=6) UTC. Quiet-day: empty-aggregate explicit. Empty-DB: try/except around DB open. Double-count: WHERE kind='page_view' against page_views; separate aggregate on affiliate_clicks.

7. test_abs_mobile_nav.py — hamburger menu + drawer on every public page

Failure shape: #menuBtn not found in HTML, #mobileMenu drawer not found, #mobileMenu missing nav item 'X', #mobileMenu is not hidden by default, /scripts/app.js 404, or cache-bust ?v=<version> missing.

Cheap-first check: curl -fsSL <url> | grep -E 'id="menuBtn"|id="mobileMenu"|id="mobileBackdrop"'. Empty grep → markup regressed. Cache-bust is a separate bug class — without it CF caches /scripts/* for 4h.

Canonical fix: Restore the three IDs in src/layouts/Base.astro. Update EXPECTED_NAV_ITEMS if you renamed/added a nav item. Default drawer needs hidden. Cache-bust: bump the CB constant and rebuild.

8. test_abs_analytics_backend.py — backend health + watchdog cron state

Failure shape (today, 2026-07-14): WATCHDOG DISABLED: abs-backend-watchdog job is paused and WATCHDOG STATE: state='paused' (expected 'scheduled') — the only red across the suite right now. Other shapes: HEALTHZ HTTP 0 (connection refused), watchdog script not executable, or run.sh is missing ps -o state= zombie check.

Cheap-first check: Three commands isolate which of the four sub-checks is failing — backend up, watchdog scheduled, source-level zombie check still present:

curl -fsS http://127.0.0.1:8793/healthz -w '\nHTTP %{http_code}\n'
python3 -c "import json; j=json.load(open('/opt/data/cron/jobs.json'))['jobs']; \
  w=[x for x in j if x['script'].endswith('abs_backend_watchdog.sh')]; \
  print(w[0]['state'], w[0].get('enabled'))"
grep -E 'ps -o state=|\[\[ "\$STATE"' /opt/data/abs/backend/run.sh

Canonical fix: Bring backend up with /opt/data/abs/backend/run.sh if HEALTHZ is dead. Resume the cron job (hermes cron resume abs-backend-watchdog) for today’s regression — both ✗ flip to ✅. Restore ps -o state= block and [[ "$STATE" != "Z" && "$STATE" != "X" ]] guard if the source-level check regressed.

Today’s action item: resume abs-backend-watchdog before the next deploy. Backend is healthy (49 pageviews/last hour per the PIPELINE ALIVE check); the safety net is down. The next zombie would again be invisible for hours.

Run-order and wiring

Deploy-gate order: search_bar (build output, fastest) → search_filter (Playwright, slowest) → css_load → backend_security → cloudflare_baseline → daily_briefing (offline) → mobile_nav → analytics_backend. Any non-zero exit blocks the push — no “warn but proceed” mode. Rule: when a test goes red, satisfy the test, do not loosen it.

When all eight pass but the site is still broken

A bug class not yet encoded (write the failing test THEN fix the code), a live-data-only failure (daily_briefing and analytics_backend touch the SQLite DB), or a regression on a path the suite does not cover (book pages, the matrix dashboard, /admin templates, individual review posts are sampled). Add the missing URL to the relevant script and re-run. >= N minimum-match counts in test_abs_search_filter.py are the only tunable in the suite; retune against the new corpus, do not lower.

Source files

Script	Targets	Lines
test_abs_search_bar.py	/srv/abs-site/dist/	313
test_abs_search_filter.py	live site (Playwright)	215
test_abs_css_load.py	live site (Playwright)	93
test_abs_backend_security.py	live HTTPS	491
test_abs_cloudflare_baseline.py	CF API + YAMLs	283
test_abs_daily_briefing.py	module + SQLite	421
test_abs_mobile_nav.py	live HTTPS	267
test_abs_analytics_backend.py	local + filesystem + cron JSON	296

Total: 2,379 lines, eight scripts, one deploy gate. Each failure shape in this guide is reproduced verbatim from at least one of those scripts; update this guide when a script’s failure message changes.

Sources

• /opt/data/skills/web/regression-test-builder/SKILL.md — the five-step encode-once rule and grow-the-suite discipline.
• /guides/hermes-agent-setup-v2/ — companion: layered install-and-verify flow.
• Per-script source files at /opt/data/scripts/test_abs_*.py.

Last verified: 2026-07-14 against python3 /opt/data/scripts/test_abs_*.py for all eight scripts and the live /healthz, CF tunnel, and cron state of this install. Re-run the whole suite before following a step whose environment may have shifted.