How to Roll Back a Failed Deployment: A Practical Runbook for CI/CD Teams

A rollback is not a failure of process; it's part of the process. Still, the moment a bad release reaches production is the worst possible time to invent a recovery plan. Teams that recover in minutes tend to share one habit: they wrote the steps down, rehearsed them on a quiet Tuesday, and gave a single named person the authority to say "revert." What follows is a runbook you can adapt — covering the deploy itself, database migrations, feature flags, and the point where an incident becomes a postmortem.

Decide the failure mode before you touch anything

Two questions settle most of the argument. Is the previous version actually available to redeploy, and can a fix reach production faster than a revert can? Rolling back means restoring the last artifact that ran cleanly in production. Rolling forward means shipping a corrected version. Roll forward only when the fix is small, well understood, and already reviewed — a flipped config value, a missing environment variable, a typo in a template. Roll back first when you're looking at authentication failures, data corruption, memory growth, or anything you can't explain.

Be precise about "known good," too. It's the last artifact that passed smoke tests against production traffic, not simply the newest tag in the registry or whatever sits on the main branch. Write that definition down. During an incident, ambiguity costs more time than the revert itself.

The rollback runbook, step by step

1. Declare the incident and name a decision owner. One person decides roll back or roll forward. Everyone else stops merging and stops starting new deploys.
2. Freeze the pipeline. Pause automatic promotion, block merges to the main branch, and stop any canary that's still stepping up.
3. Capture evidence before you erase it. Save logs, dashboard snapshots, failing request IDs, error samples, and the exact version hash that's live. A rollback destroys the scene of the crime.
4. Announce in the incident channel. State what's broken, what version is live, and what version is coming back. Keep it to one channel so the timeline stays readable.
5. Redeploy the last known-good artifact through the same mechanism as a normal release. Pin the image tag or commit SHA in your pipeline rather than editing pods or servers by hand; hand edits are how environments drift.
6. Deal with the database using the rules in the next section. Do this before verification, not after.
7. Pull kill switches for anything the code revert can't reach quickly.
8. Verify, then communicate. Confirm the failing flow works, then tell stakeholders and support what changed and when.

If that whole sequence takes longer than fifteen minutes on a service you own, that's worth knowing before an incident happens, not during one.

Database migrations are the hard part

Application artifacts revert cleanly. Data does not. A schema change only the new code understands turns a five-minute rollback into a restoration project. The way out is the expand-and-contract pattern, where every migration stays compatible with both the previous and the new application version.

• Expand: add the new column or table as nullable, with nothing depending on it yet. Deploy it on its own if you can.
• Migrate: deploy code that writes both the old and the new shapes and still reads the old one. Backfill existing rows in batches.
• Switch: move reads to the new shape once the backfill is verified.
• Contract: drop the old column in a later release, well after the previous app version is no longer deployable.

Two habits make this work in practice. First, never ship a destructive migration in the same release as the code that depends on it — the rollback path dies with it. Second, use a migration tool that tracks versions in both directions (Flyway, Liquibase, Alembic, Rails migrations and similar), and actually test the down migration in staging before you need it.

Some changes can't be reversed at all: dropping a column, rewriting values in place, deleting rows. For those, take a verified snapshot first, note the exact timestamp in the incident channel, and add a line to the runbook: "If this release included migration X, the app cannot roll back below version Y — escalate." When data has already been written in the new format, accept that you're running a partial rollback and plan the forward fix.

Feature flags: the cheapest rollback you own

Flipping a flag takes seconds, ships no artifact, and works when code can't be reverted — a mobile client already in users' hands, an API contract other teams depend on, a change spread across several services. A few rules keep it useful:

• Give every flag an owner and an expiry date. Stale flags become their own failures.
• Separate release flags (temporary, deleted after full rollout) from operational kill switches (permanent, documented).
• Test the off state. A kill switch nobody has ever pulled is a guess, not a control.
• Make flag changes audited and instantly reversible. Nobody should be flipping five flags from memory at 3 a.m.

Verify that the rollback actually worked

A green pipeline is not verification. Exercise the failing flow directly, then compare error rate and latency against the same period last week rather than the last five minutes. After that, check the edges people forget:

• Queues and scheduled jobs. The bad version may have enqueued work the old code cannot parse. Drain, replay, or dead-letter those messages deliberately.
• Caches and CDNs. Stale bundles keep running in browsers and at edge nodes after the origin has moved on.
• Data written by the bad version. If it's invalid under the old code, you're in a partial rollback and need a data fix.
• Clients you can't revert. Mobile and desktop apps shipped to users only move forward, so your API has to tolerate older clients for as long as they're in the wild.

When a rollback should trigger a postmortem

Not every revert needs a review. These do:

• Any customer-visible outage or data loss, however brief.
• A rollback that required manual database intervention or a restore.
• An incident where deciding took longer than fixing — that points at unclear ownership or missing runbooks.
• A rollback caused by a bug existing tests should have caught.
• Two or more rollbacks of the same service in one quarter. The release process, not the release, is usually the problem.

Run these blameless and time-boxed, and track three numbers: time to detect, time to decide, time to recover. Hold a lighter review for near misses too — a rollback you almost needed is cheaper to learn from than the one you did.

Make the runbook real before you need it

Keep it in the same repository as the pipeline, so it changes when the pipeline does. Write it with roles rather than personal names, and include the actual commands, dashboard links, and artifact-retention details. A registry that keeps only the last ten images may not have the version you need to restore.

Then rehearse. Once a quarter, roll back a service in an environment that looks like production and time it. Fix what the drill exposes: unpinned tags, migrations with no down path, alerts that fire after customers notice. And make reverting a normal, unpunished action. The teams that recover fastest aren't the ones that never break production — they're the ones who agreed in advance on what to do when it happens.

Photo: StockSnap / Pixabay