Know which cell is affected
“Which cell is broken?” has two halves. On Runway, where your service runs today, there are no cells; you split by region and cluster. On Cells, the cell identifies itself on alerts, not on metrics you can query centrally. Set up the first half now so the second half has something to attach to.
Today, on Runway
1. Get SLO alerts
Register your service in the runbooks metrics catalog using the
runway-k8s-archetype. You get apdex and error-ratio SLOs (default target
0.999), traffic-absence alerts, and CPU, memory and HPA saturation alerts,
plus a generated dashboard at
https://dashboards.gitlab.net/d/<runway_service_id>-main. Steps are in
Dashboards, SLOs, and alerts
and the
Runway observability reference.
Alerts route through Alertmanager to Slack and incident.io by the team you
set in the catalog entry. Only S1 and S2 alerts page the on-call SRE, and
those severities need a production readiness review. Set your own channel;
the default feed is described as very noisy.
2. Split by where it runs
Runway series carry env (staging or production), cloud_region,
k8s_cluster_name (for example runway-gke-gprd-us-east1), cloud_provider
and cloud_runtime. Put cloud_region and k8s_cluster_name in your
dashboard’s by clauses so a single-region problem is visible as such.
Example:
sum by (k8s_cluster_name, cloud_region) (
rate(http_requests_total{env="production", k8s_namespace_name="<runway_service_id>", code=~"5.."}[5m])
)Application metrics carry k8s_namespace_name, as in
Query your metrics;
kube-state metrics such as the restart counter use plain namespace. The
archetype’s own split labels are backend_name, response_code_class and
cloud_region. There is no cell label on Runway metrics.
3. Your own SLIs
If the load-balancer SLIs are not enough, Metropolis generates recording
rules, burn-rate alerts and an overview dashboard from a
GitLabServiceMetrics resource, split by the labels you name. It is in
sandbox and not yet on the paved path, see
Ship service metrics with Metropolis.
On Cells
How a cell identifies itself
Each cell runs its own Prometheus and Grafana. The cell’s Prometheus
remote-writes alert series only to the global Mimir; cell metrics stay in
the cell
(
Observability 2.0,
ADR 028, observability federation).
Alerts arrive at the global Alertmanager with tenant=<tenant_id> and
tenant_type=cell, stamped by the cell’s monitoring chart, and are grouped
by alertname, tenant, tenant_type
(
runbooks MR 10191).
So the cell is the tenant label on the alert. Tenant ids look like
c01k21yz9qqajjfn6z.
Those alerts land in #cells_alerts today, with routing to incident.io as
the next step
(
Refine Cells alert structure).
To look at a cell’s own dashboards, open its Grafana at
grafana.<managed_domain>, or find it federated in
dashboards.gitlab.net. Tier-2 on-call for
Cells routing and topology is #g_cells_infrastructure
(
tier-2 on-call).
What this means for your service
Write your alerts and dashboards so they work when a tenant label appears:
never hard-code a cluster or region, and keep by clauses label-driven. The
pattern that makes dashboards travel with the service is in
Co-located Grafana dashboards.