Rollback Before You Deploy: Name the Trigger, Artifact, and Decision Maker

A release without a written rollback plan forces a hurried revert while the operator is reading error logs. Write one page before anyone sees the change; name the accountable decision maker.

When to run this playbook

Run it for every release that touches production traffic, a paid dependency, or an irreversible step. The bar is not “is this a big change” — the bar is “can a hurried operator run the wrong command and make it worse.” If yes, the release gets a plan. Specifically: any release that swaps a primary dependency or upgrades a framework in place; introduces a new external side effect (outbound messages, billable calls, schema writes, third-party state changes); is deployed by an automated pipeline without a human in the loop; or one you cannot reproduce the prior state of after the fact. If you cannot name the artifact you would roll back to, you have a wish, not a release.

Step-by-step procedure

A numbered sequence that produces the one-page plan. Run it in order. Do not skip because the release feels small.

1. Pin the prior known-good artifact. Pull the identifier of what is currently live — the SHA, tag, image digest, or release name — and confirm it is reachable from the rollback command on your platform. If you cannot name it, stop and fix that first.

2. Name the decision maker. One person’s name and one contact channel. If they will be unavailable during the observation window, name the deputy in the same line.

3. List the trigger conditions. Each trigger is one sentence with a measurable threshold and an evaluation window. Five triggers is plenty; twenty is a wish list. Every trigger has to be checkable from the observability you already have running. Examples: error rate over a threshold for a defined window; latency p95 over a threshold; a specific log line more than N times; an external callback failing more than M percent; a cost anomaly over a ceiling. Anything subjective is not a trigger.

4. Write the recovery command or UI path in the platform’s own language. Use the verified form, already in your platform’s documentation. Do not copy a command from a stale blog post and treat it as gospel. Sketched examples by family — replace the placeholders for your environment, and confirm each against your platform’s current docs:

   Platform family	Recovery action	Reference
   Kubernetes-style rollout	kubectl rollout undo deployment/<name> --namespace=<ns>, then kubectl rollout status deployment/<name> --namespace=<ns>	kubectl rollout
   Docker Compose services	Pin the prior image tag or digest in the compose file, then docker compose up -d. Compose ships no single “rollback” CLI.	deploy key in the Compose reference
   GitHub Actions deployments	Start a new run that explicitly ships the prior artifact. Re-running the workflow at its current head is not reverting.	Deployments and environments
   Hosting-provider consoles	A named button on the deploy detail screen — usually “rollback to previous deploy” or “promote prior release.” The plan names the screen, the button label, and the artifact displayed.	Your provider’s docs

5. Document the data and schema state. State explicitly what the revert assumes. If the release is purely additive, the assumption is “data is unchanged.” If it wrote to a datastore, name the writes and decide whether the revert rolls them back. Some writes cannot be reversed (third-party side effects, payment events); for those, the plan names the recovery step that follows the revert, not the revert itself.

6. Write the communication line. One channel. One minimum message: “Revert started. Trigger: . Time: . Reverting to: .” Operators downstream of the release are told even if the revert is invisible to them.

7. Set the observation window. A bounded duration the decision maker watches the release. Five to thirty minutes is typical. The window has a stop time and the stop time is honored. If the release is still uncertain at the end of the window, that is itself a trigger: the release is kept only by an explicit decision, not by silence.

8. Record final evidence. After the window closes, write the actual outcome to the audit trail: artifact shipped, triggers watched, whether any fired, time elapsed, decision, and the artifact now considered live. A release with no recorded outcome is one you will repeat.

The page is complete when all eight fields are filled and the recovery command has been tested at least once in a non-production environment.

Checkpoints

Step	Checkpoint
Pin the prior artifact	The identifier is written down and matches what the live system reports as currently running
Name the decision maker	One name + one channel + one deputy line; “team” is not acceptable
List trigger conditions	Each trigger is one measurable sentence; the metric is already wired
Write recovery command	The command runs in a non-production environment and returns to the prior artifact
Document data and schema state	The plan states the revert’s assumptions about data; irreversible steps are named
Write communication	A channel exists; the minimum message is templated and ready to send
Set observation window	A stop time is named; the decision maker is present for the window
Record final evidence	The audit line is written before the next release starts

A checkpoint that cannot be passed is a release that cannot ship.

Rollback and recovery

A rollback is not always the right recovery. The plan names three classes of failure and the recovery each one takes.

• Revertable. The release introduced a regression that does not depend on data the release wrote. A trigger fires, the recovery command runs, the prior artifact is confirmed live, the window closes, the audit line is written.
• Partially revertable. The release changed data or schema alongside code. The revert covers the code; a separate data-recovery step covers the data, listed as a hard prerequisite to the code revert. If the data step is not safe to run, the release is not revertable and the plan names the alternative.
• Not revertable. The release produced irreversible external side effects — outbound messages, billable calls, third-party state. A blind revert may double-send, double-charge, or leave a third-party system inconsistent. The recovery is an explicit forward-fix: identify the side effects, remediate them in the systems that received them, then decide whether the code is also rolled back. The plan names this case before the release and refuses to call it “rollback.”

A security incident is its own category: stop the release, do not revert, do not communicate via the affected channel, follow the security runbook.

The plan is a guardrail, not a script. If a trigger fires, the decision maker runs the recovery command; the decision maker does not negotiate with the page. If the recovery command does not work, escalate to the platform operator, not improvise a second revert.

Variations

• No-code / UI-only releases. The recovery command is a click sequence. The plan names the screen, the buttons, and the artifact the screen claims to revert to.
• Agent-driven releases. An automated pipeline is the releaser; the decision maker is still a named human. The plan names the kill-switch (stop file, database row, environment variable, webhook) and the order the human can flip it. The pipeline never rolls itself back without the decision maker’s signal; it stops and waits.
• Multi-service releases. Each service has its own prior artifact and recovery command. The plan orders reverts by blast radius (smallest user-facing impact first) and names the data reconciliation step that runs after all reverts are confirmed.
• Long-running migrations. Data writes are irreversible during the migration window. The plan names the migration as a separate phase with its own rollback; the application-code rollback assumes the migration completed and adapts to the new schema. Two plans, two windows.

Anti-patterns

• “We can roll back from git.” Git is a history of edits, not a deployable artifact. If your platform does not surface the prior deployable from git, the “rollback” is a manual rebuild.
• “Triggers are whatever shows up in the dashboard.” A dashboard is a surface for human judgment, not a trigger. Triggers are evaluated by the platform or by a script.
• “We’ll write the rollback command after the release.” The release is the worst time to write a command you have not tested.
• “Revert is safe, just run it.” Revert is safe only when the release did not write data the revert would leave inconsistent. The plan names the boundary, not a hope.
• “We just don’t ship failures.” Every release ships surprises. The plan is what makes the surprise a five-minute reversal instead of a four-hour outage.
• “The decision is whoever is on call.” On-call rotation is a coverage schedule, not a decision maker. The plan names a person for the window.

Done means

A release is done when the plan is written, dated, and acknowledged by the decision maker; the recovery command has been tested in a non-production environment; the observation window has elapsed with no trigger fired, or a trigger fired and the recovery was completed and audited; the final evidence line is written; and the plan plus evidence are stored where the next release’s plan can reference them.

What this article does NOT cover

• Choosing an observability backend. The plan assumes you already have metrics, logs, and traces that surface the trigger conditions.
• Deciding whether a release should happen at all. This playbook assumes the release has already passed whatever review your team uses.
• Security incident response. A security event during a release is not a rollback; it is a separate runbook.
• Schema-migration mechanics. Migrations are referenced as data-side steps; the playbook does not design the migration itself.
• Universal commands. The recovery command is platform-specific and must be verified against your platform’s current documentation.

Related guides

• Why every agent needs a cost cap on day one — the cheapest guardrail that turns a runaway release into a tripped circuit breaker.
• The 1-line observability hook that fits any agent — the minimum signal you need to evaluate the trigger conditions this plan lists.
• Agents and the deploy gate: when they push — what an automated releaser must surface before it counts as a deploy.

Sources

• Google SRE Workbook — Canarying releases
• Kubernetes — kubectl rollout
• Docker — Compose deploy reference
• GitHub Actions — Deployments and environments
• Microsoft Azure Well-Architected — Safe deployments