Run database migrations with migrator

Migrator is the one way a Theseus service defines, validates, applies, and rolls back Postgres schema migrations. Your repository supplies SQL files in a fixed layout; a hook in your chart runs them before each release starts, with an explicit pass or fail. This page is the task reference. If you have never used it, start with the tutorial; for the full contract, the README is the source of truth.

All commands run from the published image, so nothing is installed locally. The image is published for linux/amd64 and linux/arm64, so it runs natively on Apple Silicon as well as on CI runners:

export MIGRATOR=registry.gitlab.com/gitlab-org/theseus/migrator:v1

Author a migration

docker run --rm -v "$PWD/db/migrations:/migrations" $MIGRATOR new <name>

<name> matches ^[a-z][a-z0-9_]*$. The command writes db/migrations/<version>_<name>.sql and db/migrations/down/<version>_<name>.down.sql and prints both paths. Fill in the forward SQL by hand or from any schema tool you already use. Write the down SQL yourself.

Rules that apply to every file:

• Each file runs in one transaction. Statements that cannot run inside a transaction block fail: CREATE INDEX CONCURRENTLY, VACUUM, and ALTER TYPE ... ADD VALUE on older Postgres. Split them out or wait for the planned per-file directive.
• Session guards are fixed at lock_timeout=10s and statement_timeout=15min. A migration that waits longer than ten seconds for a lock aborts and the run resumes on the next attempt.
• Applied migrations are immutable. Editing one fails the next up with has changed since it was applied. Write a new migration instead.
• Versions only increase. A file whose version sorts before an already applied one is rejected; status lists it under out_of_order. If two branches merged in the wrong order, recreate the newer file with migrator new so it sorts last.

Mark a migration irreversible

Leave the .down.sql with no SQL statements: empty, whitespace, or comments only. The file must still exist so the choice is visible in review. down refuses to cross an irreversible migration unless told to skip it (see Roll back).

Validate in CI

The migrator image is distroless, so it cannot be a CI job’s image: directly. Build your derived image in the job and drive it with Docker, the same way migrator’s own smoke test does. This also proves the artifact you will ship, not just the files in the checkout.

validate:migrations:
  stage: validate
  image: docker:cli
  services: [docker:dind]
  variables:
    DOCKER_TLS_CERTDIR: ""
    DOCKER_HOST: tcp://docker:2375
  script:
    - docker build -f db/Dockerfile -t migrations:ci .
    - docker run --rm migrations:ci validate
  rules:
    - changes: [db/migrations/**/*, db/Dockerfile]

For a stronger check, run the round trip against a throwaway Postgres so the SQL is proven to apply and to reverse:

roundtrip:migrations:
  stage: validate
  image: docker:cli
  services: [docker:dind]
  variables:
    DOCKER_TLS_CERTDIR: ""
    DOCKER_HOST: tcp://docker:2375
    DSN: postgres://app:app@pg:5432/app?sslmode=disable
  script:
    - docker build -f db/Dockerfile -t migrations:ci .
    - docker network create ci
    - docker run -d --name pg --network ci -e POSTGRES_USER=app -e POSTGRES_PASSWORD=app -e POSTGRES_DB=app postgres:16
    - until docker exec pg pg_isready -U app -d app; do sleep 1; done
    - docker run --rm --network ci -e DATABASE_URL=$DSN migrations:ci up
    - docker run --rm --network ci -e DATABASE_URL=$DSN migrations:ci down --to-version 0
    - docker run --rm --network ci -e DATABASE_URL=$DSN migrations:ci up
  rules:
    - changes: [db/migrations/**/*, db/Dockerfile]

Bind-mounting db/migrations/ into docker run does not work under docker-in-docker, because the path lives in the job container, not on the daemon’s host. Building the derived image sidesteps that.

The migrator component in common-ci-tasks packages these two jobs, plus the release-tagged image build below, as validate_migrations, roundtrip_migrations and build_migrations. Prefer including it over copying the YAML.

Tests and CI jobs without Docker

A job that runs against a Postgres service, such as a Go test job, has no Docker daemon and cannot run the migrations image. Every migrator release from v1.9.0 carries the CLI and its engine binary as per-platform tarballs (see “Installing the CLI without Docker” in the migrator README). In CI, use the same component’s .install_migrator step instead of downloading by hand: it installs the pinned release’s binaries onto PATH with the checksum verified, so your tests can exec migrator up, down --to-version 0 and status against the test database.

include:
  - component: $CI_SERVER_FQDN/gitlab-com/gl-infra/common-ci-tasks/migrator@<version>

test:
  image: golang:1.26
  services:
    - postgres:18
  before_script:
    - !reference [.install_migrator, before_script]
  cache:
    - !reference [.install_migrator, cache]
  script:
    - go test ./...

See the component docs for the inputs and the images it supports.

Build and publish the derived image

Migrations reach production as an image layered on the migrator base, tagged with the same release as the service image.

db/Dockerfile:

FROM registry.gitlab.com/gitlab-org/theseus/migrator:v1
COPY db/migrations/ /migrations/

The base image is a manifest list for linux/amd64 and linux/arm64, so these two lines build a native image everywhere: CI, production, and an arm64 development cluster (Apple Silicon kind, k3s, or Caproni). There is no need to build migrator from source for any platform. Pin the major tag: v1 is the contract version and every v1.x.y release moves it. Releases before v1.2.0 were amd64-only, so a derived image pinned to one of those tags will not run on arm64.

A second build job alongside the existing one, using the shared .docker_buildx_base:

build:migrations:
  stage: release
  extends: [.docker_buildx_base]
  needs:
    - job: "📝 [.runway/fairway.yaml] Helm chart versions"
      artifacts: true
  variables:
    DOCKER_BUILD_FILE: db/Dockerfile
  before_script:
    - test -n "$APP_VERSION"
    - export DOCKER_DESTINATION="${CI_REGISTRY_IMAGE}/migrations:${APP_VERSION}"

The tag matters: the chart hook resolves the image at .Chart.AppVersion, the release tag, so the migrations image must carry that tag — ${APP_VERSION} from the chart-versions job, not ${CI_COMMIT_SHORT_SHA}. Tagged this way the pair is atomic: the release that introduced a migration is the release that carries its .down.sql. The Go service template’s build:migrations job is the working reference.

Run migrations before the app starts

Migrations run as a Helm hook, annotated pre-upgrade,post-install, shipped inside your chart. On every upgrade Helm runs the Job before the new Deployment is applied; a non-zero exit fails the upgrade and the previous release keeps serving. On the chart’s very first install the Job runs after the release’s resources are applied — the ConfigMap and secret it mounts do not exist until then — retrying (backoffLimit: 6) while platform infrastructure finishes bootstrapping. The Job mounts exactly what Fairway mounts into the app container, the infrastructure ConfigMap, the database secret, and a writable /tmp, so no separate credentials exist anywhere.

Fairway leaves chart/templates/custom/ alone when it regenerates the chart, and the helm-chart CI component packages whatever is in it. The template lives in the migrator repository as contrib/fairway/migrate-job.yaml and is designed to be copied unchanged: it uses no service-named helpers, and resolves the database secret the same way Fairway’s own templates do.

mkdir -p chart/templates/custom
curl -sSfL -o chart/templates/custom/migrate-job.yaml \
  https://gitlab.com/gitlab-org/theseus/migrator/-/raw/main/contrib/fairway/migrate-job.yaml

Then name the migrations image in your manifest, under the custom key Fairway leaves open. This is the one value the template requires, and it fails with a clear message if it is missing:

# .runway/fairway.yaml
spec:
  values:
    custom:
      migrations:
        repository: registry.gitlab.com/<group>/<service>/migrations

The image tag defaults to .Chart.AppVersion, which the helm-chart component sets to the release tag. Tag the migrations image the same way (see Build and publish the derived image) and the two always travel together.

Optional values under custom.migrations: tag overrides the image tag, baselineVersion sets BASELINE_VERSION for a first deploy against an existing schema (see Adopt a database that already has a schema), and logLevel sets GITLAB_LOG_LEVEL.

Check the rendering before you push. With your generated chart in chart/:

helm template svc ./chart --show-only templates/custom/migrate-job.yaml \
  --set infrastructure.postgresql.secretRef.name=example \
  --set custom.migrations.repository=registry.example/svc/migrations

Migrator’s own pipeline renders this template against a freshly generated Fairway chart on every change, so a Fairway release that renames a volume is caught there rather than in your deploy.

Remove any schema-at-startup code from the service. Two writers to the same schema is the failure mode this replaces.

Where this is heading

The hook is the working form today. The destination is a one-line declaration in the manifest, with Fairway rendering the mounts and run-once semantics itself:

spec:
  values:
    initContainers:
      - name: migrate
        image:
          repository: registry.gitlab.com/<group>/<service>/migrations
          tag: <release-tag>
        args: ["up"]

Fairway already accepts that block, but it does not yet give an init container the infrastructure mounts or run-once behaviour ( fairway#49). When it does, migrating is deleting the hook template and adding the block above; the migrations image and db/migrations/ directory do not change.

Adopt a database that already has a schema

A service migrating from schema-at-startup has tables but no migration history. up refuses that database by default because it usually means the wrong connection string:

database already contains schema objects but no migration history

Set BASELINE_VERSION to the version that matches the existing schema. Migrator records it as applied without running it, then applies anything newer:

# .runway/fairway.yaml
spec:
  values:
    custom:
      migrations:
        baselineVersion: "20260928101500"

The hook template turns that into BASELINE_VERSION on the Job. Remove the value after the first successful deploy. status reports the versions the baseline covers in a baselined list, separate from pending; they will never run.

If the database really is meant to be taken over as is, with no matching migration, ALLOW_UNMANAGED_DATABASE=true applies everything from the start. Reach for the baseline first.

Roll back a release

Rollback is deliberate: migrator never runs backward migrations on its own. The procedure is a one-off Kubernetes Job from the newer image, because only the release that introduced a migration carries its .down.sql:

1. Run status from the N+1 image and confirm the current version.
2. Run down --to-version <V> from the N+1 image, where V is the last version release N ships. Each step is its own transaction, newest first; a failure mid-way leaves a consistent state and the same Job re-run resumes.
3. Verify with status, then deploy release N. Its migration hook finds nothing pending.

Additive migrations rarely need this; if release N tolerates the newer schema, just deploy N. The rollback runbook has the Job manifest, the --skip-irreversible semantics, and the mid-rollback failure modes.

Read the result

Every run ends with exactly one JSON line, "msg":"migrator finished", on stderr:

Exit codes:

Because these are LabKit structured logs, they land in the same place as the service’s own logs, queryable by command and outcome — for a successful run, that is where to look, because Helm deletes a succeeded hook Job immediately (hook-delete-policy: hook-succeeded). A failed Job is kept for up to a day (ttlSecondsAfterFinished), or until the next release replaces it, so kubectl logs job/<release>-migrate works while you debug the failure.

Current limitations

• A hook, not an init container. Fairway supports declaring init containers, but it does not yet mount the infrastructure config, the database secret, or a writable /tmp into them, and it renders them as sidecars rather than run-once containers. That is fairway#49. Until it lands, the pre-upgrade hook above is the supported form. One consequence: a hook runs before the release’s ConfigMaps are updated, so it reads the previous release’s infrastructure config. That is fine unless a release changes the database binding itself.
• First installs migrate after the pods start. On the chart’s first install the hook runs post-install — pre-install cannot work, because the ConfigMap and secret the Job mounts are not applied until the install’s resources land — so there is a short window where the app’s pods are up against an unmigrated database. The generated chart’s probes do not depend on the schema, and the Job retries (backoffLimit: 6) while platform infrastructure (the database instance and its secret) finishes bootstrapping.
• One database per service. Fairway and Runway do not yet provision or deliver more than one Postgres dependency per service, so migrator’s multi-database layout is not documented here until they do.
• No up --to-version. up always applies every pending migration.
• Lock contention reads as an engine failure. When several replicas start at once, waiting replicas may exit 2 with migration engine failed to start. Kubernetes restarts them and they succeed. A clearer message is planned.

The full list, with the follow-up for each, is in the README.

Related

• Add database migrations to your service, the walkthrough
• Decision: database migrations ship as a forward/backward contract, the agreement this implements
• Theseus: Database Migration epic, the roadmap including Milestone 2, background jobs and backfills