Check pod health: restarts and probes

Restarts and probe failures are the same investigation. A rising restart count means the kubelet killed a container, usually an out-of-memory kill or a failed liveness probe. The probe events on the pod tell you which.

Runway does not ship a restart or probe dashboard. Its default alerts cover apdex, error ratio, traffic absence, and CPU, memory and HPA saturation ( Runway observability reference). The metrics exist in Mimir; you query them yourself.

Today, on Runway

1. Count restarts

In Grafana, pick the Mimir - Runway datasource and run:

# Restarts per pod over the last hour. Namespace is your runway_service_id.
sum by (pod) (
  increase(kube_pod_container_status_restarts_total{namespace="<runway_service_id>"}[1h])
)

Increases indicate OOM kills or liveness probe failures. Add k8s_cluster_name or cloud_region to the by clause to see which cluster is restarting. The Feature Flags team’s overview dashboard in the runbooks repository uses this query, so it is a known-good pattern.

Or, with read-only kubectl:

kubectl -n <runway_service_id> get pods
# RESTARTS column; "(3m ago)" shows the most recent restart.

2. See why a pod restarted

kubectl -n <runway_service_id> describe pod <pod-name>

Read two places. Last State on the container shows Reason: OOMKilled or Reason: Error with the exit code. Events at the bottom shows Liveness probe failed or Readiness probe failed with the HTTP status or connection error. Readiness failures do not restart the pod; they pull it out of the Service, so you see them as lost traffic rather than in the restart count.

Container logs carry a restart-count attribute in ClickHouse. In Grafana, datasource ClickHouse - Runway Production, table observability.otel_logs, filter ServiceName = '<runway_service_id>'. See View your logs.

3. Check the probes directly

LabKit v2 serves /-/liveness and /-/readiness on the probe port, :9090, next to /-/metrics ( Get Prometheus metrics on Runway). Readiness includes any checks you registered with AddReadinessCheck, such as the database ( LabKit).

Unverified. kubectl port-forward needs create on pods/portforward, which a plain read-only role may not grant; the Runway cluster-access page promises only that apply, delete and edit are denied. If the command below is forbidden, the probe events in step 2 are your read-only path. Confirm with Runway in #g_runway.

kubectl -n <runway_service_id> port-forward <pod-name> 9090:9090 &
curl -si localhost:9090/-/liveness
curl -si localhost:9090/-/readiness

A non-200 readiness response body names the failing check.

Fairway wires the Deployment’s probes from your manifest. The paths and timings are under spec.values.*Probe in .runway/fairway.yaml, with floors of timeoutSeconds ≥ 2, periodSeconds ≥ 5, failureThreshold ≥ 2 ( Fairway service owner guide). Older Fairway versions rendered identical probes for gRPC services ( fairway#64, fixed 2026-09-16); if your gRPC pods restart when a dependency is slow, update Fairway and set a distinct readinessProbe.service.

4. When you need more than read access

Deleting a stuck pod or restarting a Deployment is a write operation, which your kubeconfig denies. Ask in #g_runway with the namespace and pod name. For a service-wide problem, roll back instead, see Roll back a deployment.

On Cells

Each cell runs its own Prometheus and Grafana. Cell metrics are not remote-written to the central Mimir; only alert series are ( Cells observability federation, ADR 028). So the restart query above, run against Mimir - Runway, will never show a cell. The per-cell Grafana is at grafana.<managed_domain> for that cell and is federated into dashboards.gitlab.net.

Not yet available. There is no modular-service deployment on a cell to inspect yet, and the Cells observability stack is still being validated. Tracked in Test Cells Observability. Owner: Tenant Scale, Tenant Services. Ask in #g_cells_infrastructure.

Operations Observability Cells

Last updated on October 6, 2026

See what is running in production Roll a deployment