Split Your Monitor From Your Worker: Reliable Scheduled Agent Jobs.

A scheduled job that quietly does the wrong thing is worse than one that loudly fails. Give the scheduler one job — decide that work is due — and the worker a different job — own a single, idempotent unit and finish it. When the same process both decides and does the work, a missed trigger and a duplicate trigger look identical to the operator. Splitting the two makes both visible and recoverable.

The playbook is platform-neutral. The commands belong to your scheduler and your runtime; the boundaries between the monitor, the scheduler, the worker, and the durable record do not.

When to run this playbook

Use this playbook when a scheduled agent can repeat a side effect, miss a required run, or run without an independent health check. It is especially useful when one process schedules and performs the work.

If the job is fully read-only, fully reversible, and already verified independently, the split is optional. Most scheduled agent work is none of those.

Step-by-step procedure

A repeatable procedure for a durable scheduled job. Reuse the same shape for subsequent jobs.

1. Name the four records before you write any code. Work inventory: units that are due, written by the scheduler. Lease: a temporary claim that a worker owns one unit. Completion record: an append-only entry proving the unit finished and what its effect was. Health signal: a heartbeat that an independent monitor evaluates against an expected window, separate from per-unit completion. If any are missing, the split is incomplete.
2. Make the scheduler thin. It translates time into units. It checks the schedule, identifies the unit that is due, writes a work-inventory row, and exits. It does not call the external API, write to the destination system, or know how the work is done. A scheduler that performs the work is a worker that happens to wake up on a clock.
3. Make the worker the owner of the unit. The worker reads the inventory, takes a lease, performs the work, and writes the completion record before releasing the lease. If the worker fails, the lease expires; another worker, or the same one on its next attempt, picks the unit up. The worker decides whether a run is a duplicate by reading the completion record, not by trusting the scheduler to be quiet.
4. Give the worker an idempotency key for every external effect. Every call with a side effect outside your system gets a stable key derived from the unit id. The key travels with the request; the receiving system uses it to deduplicate. The key is the worker’s evidence that the effect has already happened or has not — independent of whether this is the first or the fifth attempt.
5. Use an independent monitor for alerts. The monitor evaluates the scheduler’s inventory and the worker’s completion record against the expected window; it does not rely on a success message from the worker. A worker that fails after completing its unit does not page; a unit that remains incomplete past its expected window does. The interrupt channel is reserved for conditions that need a decision, not routine success.
6. Persist the completion record before the worker exits. It is the durable proof that the effect was applied. It is the last step, after the side effect and before the lease release. A worker that crashes between the effect and the record may double-run; the receiving system absorbs that with the idempotency key, and the next worker discovers the work is already done and records the duplicate.
7. Treat the scheduler, the worker, and the monitor as three independent services. Each can be restarted without taking the others down, scaled independently, and replaced. If any depend on the in-memory state of another, the split is not real.
8. Verify the split with a forced duplicate and a forced silence. Fire the scheduler twice for the same unit; the external effect happens once and the completion record reflects the duplicate attempt. Then kill the worker mid-run; the lease expires, the next run picks the unit up, the idempotency key prevents a second effect, and the monitor alerts on the silence rather than the duplication.

Checkpoints

Step	Checkpoint
Four records named	Work inventory, lease, completion record, and health signal each have an owner
Scheduler is thin	The scheduler exits after writing the inventory row; no external calls in the scheduler path
Worker owns the unit	Lease taken before the effect, released after the completion record
Idempotency key on every external effect	A stable key derived from the unit id travels with every side-effecting call
Alerts separated from completion	The channel fires on missing heartbeat or failed lease, not on healthy completion
Completion record persisted first	Written after the effect and before the lease is released
Three independent services	Each role can be restarted, scaled, or replaced without taking the others down
Duplicate and silence drills pass	A forced double-trigger causes one effect; a forced kill causes an alert, not a duplication

A checkpoint that cannot be passed is a job that cannot be called reliable.

Rollback and recovery

Three failure modes cover most of what an operator will see.

• Double execution of a side effect. The idempotency key was missing or the receiver does not honor it. Add the key, then reconcile the completion record against the side-effect log. Some side effects are not safely reconcilable; those are the ones the playbook should never have run without a human decision.
• Missed run because the scheduler was down. The work-inventory row is missing for a window. Backfill it from the schedule definition; the worker takes it on the next poll. If the schedule is the source of truth, the next scheduler run after recovery produces the missing row on its own.
• Worker stuck past the lease window. Let the lease expire, let a new worker take the unit, and use the idempotency key to absorb the in-flight work. The stuck worker eventually completes, finds the completion record, and exits without writing a duplicate. The monitor alerts on the silence, not the duplication.

A scheduled job that silently re-runs can eventually bill the operator twice.

Variations

• Single-process for development, three roles in production. One process can perform all three roles with the four records still named. The split is a deployment change, not a rewrite.
• No external scheduler. A long-lived worker with an internal timer can satisfy the scheduler role as long as it writes the inventory separately from the effect and the heartbeat separately from the completion record.
• Multiple workers, one inventory. The lease is the only thing that prevents two of them from acting on the same unit. If the inventory is sharded, each worker owns a subset.
• Human-in-the-loop unit. The work inventory marks the unit as “needs review”; a reviewer approves, the worker takes the lease, the rest of the flow is unchanged, and the review is recorded in the completion record.
• Durable orchestration for long-running units. The worker can use a workflow engine that checkpoints progress and resumes from the last completed step. The idempotency key and completion record are still required; the engine is a recovery tool, not a substitute.

Anti-patterns

• “The cron entry IS the schedule.” A cron entry is a trigger, not a schedule. A schedule is a list of units that are due, with an audit trail.
• “Idempotent means the code is safe to run twice.” Idempotent means the effect of running it twice equals running it once. If the code is safe but the receiver is not, the work is not idempotent.
• “We have retries, so we are safe.” Retries without an idempotency key are duplicates without a ceiling. The receiver, not the retry policy, decides whether the second attempt is safe.
• “The alert fires on success, so we know it ran.” A success-only alert trains the operator to ignore the channel. The channel should fire on the absence of a heartbeat.
• “We will add idempotency later.” Once a duplicate has caused a side effect, the effect has already been received. Idempotency is a design decision, not a postmortem fix.
• “The scheduler and worker share a database transaction.” Then they are one role with extra steps.
• “A silent run is a healthy run.” A scheduled job that stops running is indistinguishable from one running fine, until the operator notices. The heartbeat is what makes the silence actionable.

Done means

A scheduled agent job is done when the four records exist and are owned by their roles; the scheduler writes only the inventory and the heartbeat; the worker takes a lease, performs the effect with an idempotency key, and writes the completion record before releasing the lease; the operator-facing channel fires on missing heartbeat or failed lease, not on healthy completion; the duplicate and silence drills pass; and the next unit can be triggered, completed, audited, and recovered without an operator present.

What this article does NOT cover

• Choosing a scheduler or queue. The playbook assumes you already have a trigger, a queue, or both; it does not pick one.
• Idempotency on the receiving side. The playbook demands the key; the receiver consumes it.
• Durable workflow internals. Checkpointing and replay are referenced as a recovery tool; the playbook does not design the engine.
• Cost ceilings and rate limits. A separate concern; pair this playbook with a budget the worker respects per call.
• Security review of side effects. The split makes side effects visible; the review is a separate runbook.

Related guides

• Tests passed, but the fix is not live — the five-check verification recipe the heartbeat here assumes as evidence.
• Rollback before you deploy — the recovery discipline that pairs with this split when a scheduled job goes wrong.
• Why every agent needs a cost cap on day one — the cost-side guardrail that makes a runaway worker a tripped circuit breaker.
• The 1-line observability hook that fits any agent — the minimum signal the heartbeat here is built on top of.

Sources

• Google SRE Book — Monitoring distributed systems
• Google SRE Book — Practical alerting
• Kubernetes — CronJob
• GitLab — Idempotent jobs
• AWS — Transactional outbox pattern
• Temporal — Workflow execution
• healthchecks.io — Monitoring cron jobs