Make Long Agent Jobs Resumable: Checkpoints Instead of Restarts.

A long agent job that restarts after every failure wastes completed work. Persist every completed step, restart at the first incomplete step, and design every external side effect so a repeated call is a no-op.

When to run this playbook

Run it the first time you build a job that runs more than a few minutes, may be interrupted by rate limits, errors, restarts, or operator decisions, or mutates anything an operator cannot undo with a single command. Specifically: a job that processes more items than fit in one session; a multi-step workflow whose costs scale with the steps retried after one failure; a scheduled task that may overlap or arrive twice; a one-shot run a colleague resumes while you sleep. If you cannot name the last completed step, the job is not resumable yet.

A useful distinction up front: a retry repeats one operation, typically the one that just failed; a resumable workflow records every completed step and restarts at the first incomplete one. Retry fits a single failed step in isolation. Resumable workflows fit jobs where partial progress has value, or where repeating earlier steps costs money, time, or user-visible side effects. Do not promise exactly-once execution for arbitrary effects; prefer at-least-once delivery with idempotent boundaries and a record of what already ran.

Step-by-step procedure

A numbered sequence that turns a long job into a resumable one. Run it once per job.

1. List every step and its effect. Write the job as an ordered list and, for each step, write what state it reads and what state it changes. A step that reads nothing and changes nothing is decoration; cut it. A step that reads nothing and changes something is a side effect; flag it.

2. Pick where the state lives. One table or file the worker can open, read, and write transactionally. SQL with a unique (run_id, step_key) constraint on the steps row, a key-value store with a completed_steps set, object storage whose file existence is the completion record, or a durable workflow engine. Verify the form against your storage engine’s current docs.

3. Promote a step into a checkpoint. For every step that changes external state, the completion record is written with the effect in the same transactional boundary, not after. The transactional outbox is one documented pattern; consumers track processed message IDs and tolerate duplicates.

4. Give every external side effect an idempotency key. Any call on which money, deletion, or publication depends gets a unique, stable key derived from (run_id, step_key), not from a timestamp or random value. Use it only where the receiver documents a deduplication contract; otherwise stop for reconciliation or a human decision after an ambiguous result.

5. Make the entry point idempotent. On picking up a run, the worker’s first action is “does this step already have a completion record?” If yes, skip. Idempotency keys protect external side effects; step-completion records protect the in-job state. Together they make the run safe to deliver more than once.

6. Define resume, not just retry. Retry repeats the failed step; resume skips every completed step and replays from the first incomplete one. For scheduled work, pick an overlap policy and apply it the same way to every step — concurrent runs (each step key-guarded), skip late runs (the worker logs the skip), or replace the prior run.

7. Add a heartbeat with an expected arrival window. A healthy long job pings an external monitor on a fixed cadence; the monitor alerts only on absence past a grace period. Plan separately for a slow worker and for a worker that cannot reach the monitor.

8. Separate scheduling from doing work. The scheduler places a small, idempotent state transition — “this run belongs to worker W, start at step N” — in durable storage. The worker reads that transition and acts on it. Overlap and double-arrival are normal.

9. Version the workflow definition. When the step list changes, store the version with the run and gate resume on compatibility. A worker resuming under an incompatible definition replays from a safe checkpoint. Deterministic replay is documented behavior of durable workflow engines.

10. Audit the run, not the report. After completion or failure, write an audit line that names the run, every step’s outcome, every idempotency key sent, and the final state. A one-line “succeeded” with no step list is one you will not trust.

The job is resumable when every step has a checkpoint, every external side effect has an idempotency key, the entry point is safe to deliver more than once, and the audit line names what ran.

Checkpoints

Step	Checkpoint
List every step and its effect	Reads, writes, and external side effects are named per step
Pick where the state lives	Store, access pattern, and atomicity boundary are written down
Promote step into checkpoint	Completion record written in the same transaction as the side effect, or reconciled on replay
Idempotency keys	Every external call uses a stable, run-scoped key; the receiver’s dedup contract is documented
Idempotent entry point	Restarting the worker at the same step is a no-op, not a duplicate
Resume vs retry	Overlap, skip, and replace policies are named per scheduled job
Heartbeat	A monitor watches for absence past a documented grace period
Separation of concerns	Scheduler places work; worker records completion; neither pretends for the other
Versioning	The run stores its workflow version; incompatible resumes replay from a safe checkpoint
Audit	The post-run record lists steps, outcomes, and idempotency keys, not just a verdict

A checkpoint that cannot be passed is a job that cannot resume.

Recovery and rollback

A resumable job has three restart postures, decided before the job runs.

• Restart from the last checkpoint. Resume at the first incomplete step. The default for any interruption the worker observes itself. On (re)start, read state, not effects.
• Restart from scratch under a new run. If the workflow version changed, or an irreversibly failed effect was checkpointed, start a fresh run_id; let the old run stay frozen. Fresh keys make this a distinct attempt downstream.
• Stop and surface to a human. If a step’s effect is too expensive or irreversible to replay — deletion, paid notifications, external account changes — the worker stops, logs the in-flight step, and asks for a decision. The audit line records the stop.

A rollback of the job is not the same as a rollback of the deploy that triggered it. The deploy-rollback playbook decides whether the artifact is replaced; this one decides whether the run picks up at step N or starts anew.

Variations

• Single long worker. One process, one durable store; picks the first incomplete step, runs it, writes the checkpoint, repeats.
• Queue-driven steps. Multiple workers consume steps from a queue; the queue or its consumer deduplicates on the idempotency key.
• Durable orchestration engine. The engine handles replay, versioning, and signal handling. Each activity is the smallest unit the engine retries. Useful when partial progress is the rule.
• Scheduled overlap. A scheduled job whose interval is shorter than its duration. Pick a policy: concurrent runs (each step idempotent), one run blocks the next (others logged), or a new run replaces the old at the next checkpoint.
• Human-in-the-loop steps. Record the prompt, wait for the response, resume from the next step. The wait is a checkpoint, not a polling loop.

Anti-patterns

• “Just retry until it works.” A retry replays the step that failed; it does not invent work the partial state already produced. Retrying a long job from the top costs money, time, and side effects.
• “Idempotency keys are optional.” Not optional for any call money, deletion, or publication depends on. A timeout, crash, or duplicate worker are all legitimate causes of a second attempt.
• “Resume means start a new run.” A new run with a new run_id is a separate decision. Reaching for it on every failure throws away partial progress.
• “Silence means healthy.” A job that pings once per day or not at all is not auditable. Pair the run with a heartbeat and grace period; alert on absence.
• “The audit line is the report message.” A one-line summary is not an audit. Keep a step-by-step record whose loss would force a manual rebuild.
• “Exactly-once is the goal.” Wrong goal for arbitrary distributed work. Aim for at-least-once delivery with idempotent effects and a completion record.
• “Workflow versioning can wait.” A workflow definition will change. Stamp the version onto the run and gate resume on it; the alternative is untangling an inconsistent replay.

Done means

A long agent job is resumable when the step list is written down; each step’s reads, writes, and idempotency boundary are named; a durable store holds the completion record per step; the entry point is safe to deliver more than once; scheduled jobs have an overlap policy and a heartbeat with a grace period; the workflow version is stamped onto every run; the post-run audit names the steps, outcomes, and idempotency keys; and a manual interruption can be recovered by re-attaching to the last checkpoint without repeating completed work. A job that meets all of these stops costing more every interruption; a job that meets none stops finishing.

What this article does NOT cover

• Choosing a durable orchestration platform. A workflow engine, queue-driven consumer, and hand-rolled state machine each fit the steps above.
• Cost modeling for resumed runs. The delta between a resumed and a restarted run depends on billable calls and which steps had completed.
• Designing the prompts that drive each step. Job structure is independent of the model prompts inside it.
• Multi-tenant scheduling, fairness, or quota. Belong to the scheduler’s design, not this discipline.
• A general argument against retries. Retries are right for one operation that failed in isolation. This playbook is for the jobs retries alone won’t save.

Related guides

• Write the rollback before you deploy — partner playbook for the deploy artifact.
• Tests passed but the fix is not live — verification for the resumed run.
• Why every agent needs a cost cap on day one — completed steps shouldn’t be paid for twice.
• The 1-line observability hook that fits any agent — minimum signal for resumed-run progress.

Sources

• git-worktree
• Re-run workflows and jobs
• Monitoring distributed systems
• CronJobs
• Sidekiq idempotent jobs
• Transactional outbox pattern
• Workflow execution
• Durable orchestrations