Roll back a deployment

Roll back first, investigate second. The options below are in order of speed. Take the first one that applies.

Runway’s Kubernetes runtime has no rollback job today. The documented options are to re-run a previous deployment or revert the change ( Runway runbook). A versioned rollback is being built, see the end of this page.

Today, on Runway

1. If the change is behind a flag, turn it off

Flipping the flag to false in the feature-flags repository takes effect in about thirty seconds with no pipeline and no redeploy. See Toggle the flag. This is why new behaviour ships behind a flag that defaults to off, see Roll out new code safely.

2. Re-run the last good deployment

Open your deployment project, https://gitlab.com/gitlab-com/gl-infra/platform/runway/deployments/<runway_service_id>/-/pipelines.
Find the last pipeline whose 📤 Upload Resource Manifests: production job succeeded before the bad release. Confirm its chart version in the 🏗️ Generate Flux manifests: production job log.
Retry that pipeline’s production upload job (Retry on the job page). It re-pushes that version’s artifact as deploy:production. Flux reconciles within five minutes. The rendered manifests are job artifacts that expire after one week. If Retry fails on missing artifacts, retry 🏗️ Generate Flux manifests: [production] first; it re-renders the same chart version. If that fails too, go to option 3.
Immediately stop automation from undoing it. Otherwise the next release, or anyone retrying the bad pipeline, rolls you forward again. The Kubernetes v2 deployment pipeline honours exactly one hold: set runway_manual_production_deploy: true in your release-platform inputs and merge it as a chore: commit, which does not itself release. From then on every production upload needs a manual play ( deployment template). Tell the team not to retry the bad pipeline. For a hard stop, ask Runway in #g_runway to suspend the Flux kustomization for your service, which is the documented incident lever ( incident response).
Confirm with See what is running.

Unverified. Runway’s disaster recovery page also names RUNWAY_DEPLOYMENT_ACTION=manual and a GitLab Deploy Freeze as ways to pause deploys. At the time of writing both are implemented only in the Cloud Run deployment template; the Kubernetes v2 template reads neither, and its trigger job forwards an explicit variable list that excludes them. Do not rely on them for a Kubernetes service until Runway confirms otherwise in #g_runway.

Unverified. Re-running a previous deployment job is the procedure the Runway runbook documents, but that runbook was written for the Cloud Run runtime. The kubernetes-v2-service template also accepts a chart_version_overwrite input that pins the chart version the deployment renders, which would be a cleaner pin. Neither is documented for Kubernetes service owners. Confirm the current recommendation with the Runway team in #g_runway before relying on either during an incident.

3. Revert the merge request

Revert the offending MR on your default branch with a fix: message so it releases, and let the normal chain deploy it. This is the slowest option (a full release and deploy) but the cleanest, and it is the only one that also fixes the default branch. Follow it with See what is running.

What is coming

Not yet available. Runway is adding a versioned rollback for the Kubernetes runtime: every deploy tags the Flux bundle with a version, and a rollback retags the moving deploy:production tag from the deployment project’s Run pipeline form, without re-rendering. Tracked in Runway work item 983. Owner: Runway. When it merges, step 2 above becomes that form.

The roadmap tracks the Theseus commitments: a manual rollback process for pilot components (target Oct 16, 2026), and later a standardised automated rollback across every component (target Jan 15, 2027).

Database schema

Rolling the image back does not roll the schema back. If release N+1 ran a migration that release N cannot tolerate, run the migrator down job first, then deploy N. Additive migrations rarely need this. Procedure: Roll back a release. The migrator’s pre-upgrade hook means a failed migration never takes the old release down; the previous release keeps serving.

Flux is stuck

If pods never reach the new (or old) version, Flux may have stalled. Its state lives on a Kustomization named after your service in the runway-provisioner namespace:

kubectl -n runway-provisioner get kustomization <runway_service_id>
kubectl -n runway-provisioner describe kustomization <runway_service_id>

Unverified. The Runway cluster-access page promises only that writes are denied; whether the read-only role can read objects in runway-provisioner is not documented. If these commands are forbidden, ask Runway in #g_runway for the kustomization status.

Suspending or resuming Flux is a write operation and needs a Runway SRE. Post what you have in #g_runway and reference the Runway incident response page.

On Cells

Rollback on cells is owned by Release Managers and coordinated in #g_delivery. The tool is ringctl, whose rollback <version> command rolls a ring or a cell back to a desired version; a single broken cell is isolated by moving it to the quarantine ring, fixing it, and moving it back. Post-deployment migrations bound the rollback window: the Cells deployments design notes that cells cannot roll back without downtime once those have run, and ADR 022 scoped rollbacks out of the first ring definition.

Not yet available. There is no rollback procedure for a modular service on a cell, because there is no supported deploy path for one yet. Tracked in Delivery work item 22155. Owner: Delivery and Cells Infrastructure. Tier-2 on-call for Cells is in #g_cells_infrastructure ( tier-2 on-call).