Ship service metrics with Metropolis

This guide onboards a service team to Metropolis: you declare your service’s SLIs, SLO targets, and saturation points as a custom resource in your repository, validate the generated monitoring in CI, and ship the declaration with your chart. The operator in the cluster does the rest — recording rules, burn-rate alerts, and an overview dashboard. For why this is the direction, see the Metropolis strategy page.

This guide covers the service-team side only. Installing the operator itself is a platform concern — see Metropolis’s own docs for helm install, kube-prometheus-stack wiring, and grafana-operator delivery.

Metropolis ships in the standard Caproni rig, so you can develop against it today. Production adoption is staged across Dedicated and Cells, Runway, and Self-Managed in the Metropolis Adoption epic; Theseus-side questions are in the decision record.

Declare your service’s monitoring

Add a GitLabServiceMetrics manifest to your chart. The spec and a Metropolis CLI configuration file are the same document, so everything below works in both. A trimmed example — one SLI with a request rate, an apdex, and an error rate:

apiVersion: metrics.gitlab.com/v1alpha1
kind: GitLabServiceMetrics
metadata:
  name: my-service
spec:
  type: my-service
  tier: sv
  monitoringThresholds:
    apdexScore: 0.999
    errorRatio: 0.9995
  saturationMetrics:
    - cpu
    - disk_space
  serviceLevelIndicators:
    server:
      userImpacting: true
      significantLabels: [region]
      requestRate:
        counter:
          metric: http_requests_total
          selector:
            matchers: { job: my-service }
      apdex:
        histogram:
          metric: http_request_duration_seconds_bucket
          satisfiedThreshold: 0.5
          toleratedThreshold: 1.0
          selector:
            matchers: { job: my-service }
      errorRate:
        counter:
          metric: http_requests_total
          selector:
            matchers: { code: "5xx" }

The schema is generated from a validated proto, and the API server documents itself — kubectl explain gitlabservicemetrics.spec describes every field. saturationMetrics names points from the vendored catalog (cpu, disk_space, …); an unknown name is rejected at generation time.

Inspect what your declaration produces

The CLI renders exactly what the operator would, which is useful when you are debugging a declaration or want to understand a change before it ships:

metropolis generate --config service.yaml --output-dir out/

Treat the output as a build artifact, not source: do not commit rendered rules or dashboards to your repository. The declaration is the reviewed surface; the operator’s output is its compilation, and checking it in adds noise without adding safety. Verification happens against a running stack in the local Caproni rig, where you see the actual rules and dashboards the operator produces before anything reaches a shared environment.

Ship it with your chart

Place the manifest with your chart’s templates so it deploys with the service. The operator (installed once per cluster) reconciles it into a PrometheusRule and the overview dashboard in your namespace.

One caveat: a custom resource can only be applied where its CRD is installed — on a cluster without the Metropolis CRDs, helm install fails rather than skipping the resource. Gate the manifest behind a chart value (for example metrics.enabled) so the chart stays installable everywhere. With the CRDs present but no operator running, the resources simply sit unreconciled: missing dashboards, never broken deploys.

The intended paved path is for Fairway-generated charts to emit GitLabServiceMetrics resources consistently, and to enforce that a chart emitting one also has its PodMonitor/ServiceMonitor and metrics endpoints enabled — a declaration whose metrics are never scraped is a silent failure. Until that lands, add the manifest to your chart’s templates by hand.

Add a custom dashboard (optional)

For dashboards beyond the generated overview, ship a GitLabGrafanaDashboard with a Grafonnet entrypoint — the libraries are vendored into the operator, so the repo carries only the entrypoint:

apiVersion: metrics.gitlab.com/v1alpha1
kind: GitLabGrafanaDashboard
metadata:
  name: my-service-overview
spec:
  jsonnet: |
    local g = import 'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet';
    g.dashboard.new('My Service Overview')
    + g.dashboard.withUid('my-service-overview')
    + g.dashboard.withPanels([
      g.panel.timeSeries.new('QPS')
      + g.panel.timeSeries.queryOptions.withTargets([
        g.query.prometheus.new('$PROMETHEUS_DS', 'sum(rate(http_requests_total[5m]))'),
      ]),
    ])

Preview it the same way before it ships — render locally to inspect the JSON model, then verify it visually in the Caproni rig:

metropolis dashboard render --config dashboard.jsonnet --output-dir out/

See it locally

The standard Caproni rig runs Metropolis as an opt-in fragment, with grafana-operator and real GitLabServiceMetrics for the GitLab services the rig deploys. Enable the o11y-metrics and metropolis fragments in caproni.local.yaml (Metropolis targets the Prometheus and Grafana that o11y-metrics owns), log in to both private chart registries, and bring the rig up:

helm registry login registry.gitlab.com      # Metropolis chart and image
helm registry login registry.ops.gitlab.net  # k8s-monitoring-stack
caproni up && caproni update-etc-hosts && caproni ca trust

Apply your resource to the gitlab namespace with caproni kubectl apply, then check it reconciled and look at what it produced:

caproni kubectl -n gitlab get gsm                 # your resource: Ready
caproni kubectl -n gitlab get grafanadashboards   # delivered to the rig Grafana

Dashboards land in the gitlab folder at http://grafana.caproni.test (mise run grafana:adminpassword for the login). The Metropolis fragment doc covers the deployers, how to query the generated series directly, and tear-down.