Skip to content
The Observability Story

The Observability Story

Observability on Theseus is a property of the paved path, not a per-team project. A modular feature does not design its own metrics pipeline: it inherits one. The chain has four links — instrument once, collect automatically, render consistently, and (next) define dashboards in the repo — and each link exists because of the one before it.

Instrument once, with LabKit

Every scaffolded service ships with LabKit v2 wired in: structured logging, RED metrics, OpenTelemetry tracing, and health endpoints are present from the first commit, before a line of domain code is written.

The part that matters for the rest of this page is consistency. LabKit emits the same metric names, the same label shapes, and the same log and trace structure for every service that uses it. Two teams who have never spoken ship services whose request rates, error ratios, and latencies are measured the same way. Instrumentation stops being a convention to enforce in review and becomes a property of the library layer.

Collected with zero configuration

Deploy to Runway and the metrics are collected automatically — no scrape configuration, no annotations to remember, no per-service onboarding of the monitoring stack. The platform knows where a service’s metrics live because every service exposes them the same way.

The same holds across the observability signals:

The how-to guides above are deliberately short — that is the point. They describe how to use what is already flowing, not how to set it up.

Consistent dashboards

Uniform metrics make uniform dashboards possible. Because every service measures requests, errors, and latency identically, its dashboards follow the same shape — and anyone at GitLab can read any service’s dashboard without first learning that team’s conventions. An engineer paged onto an unfamiliar service sees the panels they expect, in the layout they expect, backed by metric names they already know.

This is the quiet payoff of the first two links in the chain: consistency at the instrumentation layer compounds into consistency at the operational layer, where it matters most — during an incident.

Where this is heading: dashboards from the service repo

The co-located dashboards strategy sets the direction: a service’s dashboards belong in the service’s repository, reviewed in the same merge request as the code they observe, shipped by the same deploy. The mechanism that completes the story is Metropolis, the GitLab Metrics Operator:

  • A service declares its SLIs, SLO targets, and saturation points in a GitLabServiceMetrics resource shipped with its chart. On deploy, the operator turns that declaration into Prometheus recording rules, burn-rate alerts, and a generated overview dashboard — built with the same metrics-catalog library that monitors GitLab.com.
  • For dashboards beyond the generated overview, a team writes a Grafonnet entrypoint in a GitLabGrafanaDashboard resource, also in the repo, also shipped with the chart. The operator evaluates it and delivers it to whatever Grafana lives in that environment — no jsonnet toolchain in the service repo, no central dashboards repository, no extra deploy.

Custom dashboards defined in the repository you are already working in, picked up automatically on deploy: the same co-location principle that governs the service’s code, configuration, and chart, extended to its observability.

Metropolis is in active development in the gl-infra sandbox and is not yet part of the paved path. The generated-dashboard and custom-dashboard flows described here are working today in that project; wiring them into the Theseus deployment path is the road ahead.

The chain, end to end

    flowchart LR
    subgraph repo ["Service repository"]
        code["Service code<br/>+ LabKit v2"]
        crd["Dashboard & SLI definitions<br/>(future: GitLabServiceMetrics,<br/>GitLabGrafanaDashboard)"]
    end
    code -->|"uniform metrics,<br/>logs, traces"| runway["Runway<br/>zero-config collection"]
    runway --> prom["Prometheus"]
    prom --> dash["Consistent dashboards"]
    crd -.->|"shipped with the chart"| metro["Metropolis operator"]
    metro -.->|"rules, alerts,<br/>generated + custom dashboards"| dash
  

The solid path is live today. The dotted path is the direction: the service repository becomes the single source of truth for what a service is — code, configuration, chart, and the dashboards that watch it.

Related reading

Last updated on