The Agent Said It Failed. Check the Work Before You Retry.

When a coding agent reports a failure, the impulse is to retry. That impulse is the trap. A failure report is one observation; the working tree, the prior run log, and the commit history are others, and they do not always agree. A retry without inspection usually produces the same failure, sometimes with a cleaner log and a misleading sense of progress.

This guide is for the moment between seeing the red FAILED banner and clicking retry. It walks through how to inspect what actually happened, how to decide whether to retry at all, and how to tell when the problem is no longer yours to solve alone.

Symptom table

The agent can fail in several recognizable ways. Match what you saw to the row before deciding what to do next.

Symptom	Most likely cause	First check
Agent reports failure, but the working tree has the expected files	The agent ran a verification step that does not match the local environment	Diff the working tree against the last commit and read the verification command’s output
Agent reports failure, working tree is empty or reverted	The agent lost state mid-run or ran in a sandbox that was discarded	Inspect reflog and any linked worktrees for evidence of in-progress work
Agent reports failure, but a follow-up commit exists after the failing run	A later repair already landed; the failure report is stale	Compare commit SHAs across the local repo and the remote
CI reports failure on the same commit after a retry	The retry used the same revision; nothing new was attempted	Read the prior run’s job log before re-retrying
Scheduled job reports failure, but downstream state shows the effect succeeded	The job ran twice or a worker re-processed the message	Search the durable record for a completed marker tied to the same unit
Job has not reported anything at all	The job is hung, the runner is down, or the alert channel is silent	Check the heartbeat or last successful completion timestamp

Decision tree

Walk this tree top to bottom. Stop at the first branch that matches.

1. Does the working tree show work the agent did not report?
   • Yes → Treat the report as stale. Inspect the diff and the prior run log before deciding. Do not retry blindly.
   • No → Continue.
2. Does the commit history include a change after the failure timestamp?
   • Yes → A repair landed. Verify the repair and stop; do not retry the old revision.
   • No → Continue.
3. Did the failure come from a verification step that depends on environment state (network, secret, tool version)?
   • Yes → Fix the environment first. Retrying without the fix will reproduce the same error.
   • No → Continue.
4. Is the failure deterministic (same input, same error, same line)?
   • Yes → The retry will not change the outcome. Open the artifact, not the retry button.
   • No → Continue.
5. Is the work idempotent (safe to repeat with the same effect)?
   • Yes → A retry is acceptable once you have recorded what you observed.
   • No → Do not retry automatically. Pause for a human decision.

If the tree ends at “open the artifact,” that is the correct answer. A retry is not a debugging tool.

Reproducible inspection recipe

Use this sequence when a coding agent or scheduled job fails. Each step is cheap and idempotent.

1. Snapshot the working tree.
   • Run git status for a porcelain inventory of HEAD, the index, and the working tree.
   • Run git diff and git diff --staged to see unstaged and staged changes since HEAD.
2. Enumerate linked worktrees.
   • Run git worktree list to confirm whether other checkouts exist for the same repository.
3. Walk recent history.
   • Run git log -n 20 --oneline for the recent commit list.
   • Run git reflog for local reference movement, including commits and resets not yet pushed. Reflog data is local and expires; treat it as a hint, not a record.
4. Inspect the prior run.
   • Open the CI run’s job log. Read the failing step’s command, its exit code, and the last successful step before it.
   • Confirm the run’s commit SHA. A retry uses the same SHA unless the commit is updated.
5. Check the durable record.
   • Search the job’s state store for a completion marker tied to the same unit of work. Look for an idempotency key, a deduplication record, or an outbox row.
6. Check the heartbeat.
   • If the job is scheduled, confirm whether a recent completion signal arrived inside its expected window. Silence past the grace period is the signal; a missing signal is not the same as a healthy job.

Write down what each step returned before you act. The point of the inspection is to make your next decision auditable, not to feel thorough.

When to escalate

Stop inspecting and ask for help when any of the following is true:

• The inspection itself keeps changing state. If a retry, a diff, or a log read mutates the working tree, treat the system as unsafe to operate on alone.
• The failure involves money, deletion, or publication. These are not retry-safe categories. They require a human decision and an explicit idempotency check.
• The retry produces a different error each time, with no pattern you can name. That is non-deterministic failure and is outside the scope of a single inspection pass.
• The provider’s documented behavior (Git, CI, scheduler, workflow engine) does not match what you are observing. Document the gap and escalate rather than guess.
• The failure has crossed a known platform boundary, such as an external service that should be idempotent but is not. The fix is rarely to retry harder.

Done means

You are done with this guide when you can answer all of the following without rereading the failure banner:

• You can name what changed in the working tree since the last good commit, or confirm that nothing changed.
• You can point to the exact line or step that failed and explain why it is not safe to retry.
• You have a recorded observation for each command you ran, so a teammate can reproduce the inspection.
• Your next action is a deliberate choice between retry, repair, escalate, or stop, and you can defend it in one sentence.

What this article does NOT cover

• Tool-specific setup for a particular coding agent, CI provider, or scheduler. Provider commands and retry semantics change; verify them against current documentation before relying on any example.
• Exactly-once execution guarantees for arbitrary distributed effects. The honest answer from the major platforms is at-least-once delivery with idempotent effects, not exactly-once.
• A full incident-response workflow. This guide assumes the failure is local, recoverable, and reversible. If it is not, treat it as an incident, not a retry decision.
• Performance, cost, or capacity tuning of CI runners, schedulers, or workflow engines. That is a separate discipline with its own evidence base.
• The internal architecture of any single agent product. Treat the agent as an opaque worker whose reports are observations to verify, not statements to trust.

Related guides

• Tests Passed, But the Fix Is Not Live — verify the running service after the work itself is proven.
• Rollback Before You Deploy — define the recovery decision before changing a live system.
• The 1-Line Observability Hook That Fits Any Agent — establish a small, independent signal before a failed report becomes your only evidence.

Sources

• Git — git-status documentation
• Git — git-diff documentation
• Git — git-worktree documentation
• Git — git-log documentation
• Git — git-reflog documentation
• GitHub Docs — Re-run workflows and jobs
• GitHub Docs — Monitor workflows
• GitLab CI — YAML reference
• Google SRE Book — Monitoring Distributed Systems
• Google SRE Book — Practical Alerting
• Kubernetes — CronJob concept
• Temporal — Workflow Execution