Operate

This section is for the engineer on call for a modular feature. Its job is to cut the time you spend inside an incident finding the right command, dashboard, or owner. Each question below has a one-line answer, the single thing to run or open right now, and a link to the detail.

Every answer is framed for a single Cell. Where the Cells path is not ready, the page says who owns it and links the tracking issue instead of inventing a procedure.

Set this up before you need it

Do these three things today. None of them can be done under pressure.

Read-only kubectl to the Runway clusters. Ask for membership of the runway-k8s-readonly group, then configure your kubeconfig through the KAS proxy on ops.gitlab.net. Write operations are denied by design.
```
glab auth login --hostname ops.gitlab.net
GITLAB_HOST=https://ops.gitlab.net glab cluster agent update-kubeconfig \
  --repo gitlab-com/gl-infra/platform/runway/kas-clusters \
  --agent <AGENT_ID> --use-context --cache-mode keyring-filesystem-fallback
```
Agent IDs are listed on the kas-clusters agents page. Source: Runway cluster access.
Grafana access at dashboards.gitlab.net to the Mimir - Runway and ClickHouse - Runway Production datasources. Open your service dashboard once so you know it exists: https://dashboards.gitlab.net/d/<runway_service_id>-main.
Your two identifiers. Your Runway service id (the runway_service_id in .runway/fairway.yaml) and your deployment project, https://gitlab.com/gitlab-com/gl-infra/platform/runway/deployments/<runway_service_id>. Bookmark the deployment project’s pipelines page. Rollback starts there.

How do I know what is running in production?

The last successful Upload Resource Manifests: production job in your deployment project is what Flux is reconciling; its Generate Flux manifests log names the chart version. Now: open https://gitlab.com/gitlab-com/gl-infra/platform/runway/deployments/<runway_service_id>/-/pipelines. Detail: See what is running in production.

How do I know how many restarts my pods have had?

Query kube_pod_container_status_restarts_total for your namespace in the Mimir - Runway datasource, or read the RESTARTS column from kubectl. Now: kubectl -n <runway_service_id> get pods. Detail: Check pod health.

How do I know the liveness and readiness health of my service?

Probe failures are events on the pod; LabKit serves the probes on port 9090. Now: kubectl -n <runway_service_id> describe pod <pod> and read Events. Detail: Check pod health.

How do I roll a deployment in production?

A feat: or fix: commit releases and deploys; production follows staging automatically unless you have made it a manual play. Now: merge the release commit, then watch the deployment project pipeline. Detail: Roll a deployment.

How do I roll back quickly in an incident?

Turn the flag off if there is one. Otherwise retry the production upload job of the last good pipeline in your deployment project, then make production deploys manual so nothing rolls you forward. Now: deployment project → Pipelines → last good pipeline → retry Upload Resource Manifests: [production]; if its artifacts have expired, retry Generate Flux manifests: [production] first. Detail: Roll back a deployment.

How do I safely roll out new code?

The Kubernetes runtime gives you rolling update and staging-first, nothing progressive. Ship dark behind a feature flag that defaults to false, then turn it on. Now: check the flag exists and defaults to off before you merge. Detail: Roll out new code safely.

How do I de-risk deployments with feature flags?

Flags are YAML in the feature-flags repository and take about thirty seconds to apply. Turning one off is the fastest rollback you have. Now: Toggle the flag. Detail: Roll out new code safely.

How can I change the application configuration for my feature in this cell?

Edit .runway/values-production.yaml for one environment or spec.values.appConfig in fairway.yaml for all, and ship it as a fix: release. A ConfigMap change alone does not restart pods. Now: decide whether you need a restart, then commit. Detail: Change configuration for a modular feature.

How can I set up monitoring so I know instantly which cell has a problem?

Register in the runbooks metrics catalog with the Runway archetype and split by cloud_region and k8s_cluster_name. On Cells, the cell is the tenant label on the alert; cell metrics never reach central Mimir. Now: open https://dashboards.gitlab.net/d/<runway_service_id>-main. Detail: Know which cell is affected.

Who owns what on Cells

Theseus produces the deployable artifact. Where it runs, how it is rolled out, and how a cell is observed belong to other teams. When a page above says “not yet available”, this is who to ask.

Team	Owns	Ask in
Tenant Scale: Cells Infrastructure	Cell provisioning, tenant model, multi-cell alerting overview, tier-2 Cells on-call	`#g_cells_infrastructure`
Tenant Scale: Tenant Services	Cells observability stack and its validation	`#g_cells_infrastructure`
Delivery and Release Managers	Ring rollouts, `ringctl` rollback, the per-cell service deploy path	`#g_delivery`
Runway	The Kubernetes runtime, deployment projects, Flux, versioned rollback	#g_runway
Feature Flags	The flags service and the per-cell flag design	the feature-flags repository
Theseus	Fairway, LabKit, the Configuration module, this documentation	#proj_theseus

For the full boundary, read Theseus and Cells: separation of concerns.

Last updated on October 6, 2026