Stella Ops Deployment Upgrade & Rollback Runbook

Audience: release engineers and platform operators promoting or rolling back a Stella Ops release. Scope: promote / roll back a release and keep the edge, stable, and airgap channels aligned.

This runbook describes how to promote a new release (Docker Compose and Offline Kit), how to roll back safely, and how to keep the edge, stable, and airgap channels aligned. All steps assume you are working from a clean checkout of the release branch/tag.

Tree-vs-doc caveat. The per-channel docker-compose.{dev,stage,prod,airgap}.yaml deployment profiles and env/{...}.env files that this runbook (and the release tooling) historically assume do not exist in the current devops/compose/ tree. The release-manifest authority, digest tooling (devops/release/*.py), alignment checker, CLI migration gate, and buildinfo probes are all real and verified; the deployment-profile layer is not. See §1.

Maintenance: update this runbook whenever the release digest inventory, release/evidence schemas, default local-model guidance, or provider-pack guidance changes.

1. Channel overview

(Two additional manifests live alongside these in devops/releases/: 2025.09-mock-dev.yaml and the machine-readable service-versions.json. They are not promotion channels.)

Compose-profile reality check (verified against devops/compose/). The full platform is run from devops/compose/docker-compose.stella-ops.yml. The infrastructure-only development stack remains devops/compose/docker-compose.dev.yml (note .yml). Release tooling and profile validation now point at files that actually exist under devops/; the historical deploy/ and devops/helm/stellaops targets are obsolete. Dedicated per-channel docker-compose.prod.yaml / stage / airgap files still do not exist, so channel promotion uses the release manifest plus the current compose stack or site-specific operator overlays.

Infrastructure components must be digest-pinned in the release manifests. PostgreSQL and Valkey are digest-pinned in the 2025.09 stable and air-gap manifests; RustFS is still tag-only (registry.stella-ops.org/stellaops/rustfs:2025.09.2) there and blocks strict production posture until release engineering publishes an authoritative digest.

Manifest digest gaps (verified against devops/releases/*.yaml, 2026-05-30). The infrastructure caveat above is not the whole story — several component images are not digest-pinned either, and one required component is missing outright:

• advisory-ai-web and advisory-ai-worker are tag-only in all three manifests (:2025.09.2, :2025.09.2-airgap, :2025.10.0-edge).
• notify-web is a required component per devops/release/check_release_manifest.py but is absent from the stable, air-gap, and edge manifests entirely, so the validator fails with missing required component 'notify-web'.
• The edge manifest additionally carries issuer-directory-web tag-only.

Consequently check_release_manifest.py does not pass against the manifests as checked in; the digest-pinned entries (Authority, Signer, Attestor, Scanner Web/Worker, Concelier, Excititor, Web UI, plus PostgreSQL and Valkey) are real, but the manifests are not yet release-publishable.

2. Pre-flight checklist

1. Refresh release manifest Pull the latest manifest for the channel you are promoting (devops/releases/<version>-<channel>.yaml).

2. Validate release authority before publication Stable and air-gap release manifests must carry the digest-pinned authority for every production image before values files are updated. This check is local-only; it does not resolve mutable tags or query registries.

   Produce the local digest artifact from build/push outputs first. Component digests must come from Docker Buildx --metadata-file outputs written by successful publish jobs; one <component>.metadata.json file is required for every component in devops/release/check_release_manifest.py. The producer can carry already digest-pinned PostgreSQL and Valkey entries from the local release manifests, but RustFS is tag-only in the current 2025.09 manifests and must be supplied by an infrastructure source captured from a real local mirror/build result.

   python devops/release/produce_image_digests.py \
       --channel-source stable=out/release/stable/artifacts/metadata \
       --channel-source airgap=out/release/airgap/artifacts/metadata \
       --manifest stable=devops/releases/2025.09-stable.yaml \
       --manifest airgap=devops/releases/2025.09-airgap.yaml \
       --infrastructure-source out/release/infrastructure-digests.json \
       --output out/release/image-digests.json
   

   The infrastructure source is local JSON with manifests.<channel>.infrastructure mappings to final image@sha256:<digest> references. It is required for any infrastructure image that is not already digest-pinned in the local release manifest. Tags, missing entries, repository/name mismatches, missing Buildx digests, and one-channel-only updates fail before any manifest is written.

   Apply the produced artifact only after the producer succeeds:

   python devops/release/apply_image_digests.py \
       --artifact out/release/image-digests.json \
       --manifest devops/releases/2025.09-stable.yaml \
       --manifest devops/releases/2025.09-airgap.yaml
   

   Artifact shape:

   {
     "manifests": {
       "stable": {
         "components": {
           "authority": "registry.stella-ops.org/stellaops/authority@sha256:<64-hex>",
           "notify-web": "registry.stella-ops.org/stellaops/notify-web@sha256:<64-hex>",
           "advisory-ai-web": "registry.stella-ops.org/stellaops/advisory-ai-web@sha256:<64-hex>",
           "advisory-ai-worker": "registry.stella-ops.org/stellaops/advisory-ai-worker@sha256:<64-hex>"
         },
         "infrastructure": {
           "postgres": "docker.io/library/postgres@sha256:<64-hex>",
           "valkey": "docker.io/valkey/valkey@sha256:<64-hex>",
           "rustfs": "registry.stella-ops.org/stellaops/rustfs@sha256:<64-hex>"
         }
       },
       "airgap": {
         "components": {
           "authority": "registry.stella-ops.org/stellaops/authority@sha256:<64-hex>",
           "notify-web": "registry.stella-ops.org/stellaops/notify-web@sha256:<64-hex>",
           "advisory-ai-web": "registry.stella-ops.org/stellaops/advisory-ai-web@sha256:<64-hex>",
           "advisory-ai-worker": "registry.stella-ops.org/stellaops/advisory-ai-worker@sha256:<64-hex>"
         },
         "infrastructure": {
           "postgres": "docker.io/library/postgres@sha256:<64-hex>",
           "valkey": "docker.io/valkey/valkey@sha256:<64-hex>",
           "rustfs": "registry.stella-ops.org/stellaops/rustfs@sha256:<64-hex>"
         }
       }
     }
   }
   

   The example abbreviates the component list for readability. The real artifact must include every component required by devops/release/check_release_manifest.py: Authority, Signer, Attestor, Scanner Web/Worker, Concelier, Excititor, Notify Web, AdvisoryAI Web/Worker, and Web UI.

   python devops/release/check_release_manifest.py \
       --manifest devops/releases/2025.09-stable.yaml \
       --manifest devops/releases/2025.09-airgap.yaml
   

   If this fails, release engineering must add the missing component metadata or replace the tag-only entry with the attested image@sha256:<digest> from the signed release build output. Current blockers (verified 2026-05-30 against the checked-in manifests): notify-web is missing entirely as a component (the validator reports missing required component 'notify-web'); advisory-ai-web and advisory-ai-worker are present but tag-only; and rustfs is tag-only in infrastructure. RustFS specifically has no local authoritative source in the current tree beyond tag-only manifest entries. (The edge manifest also carries issuer-directory-web tag-only, but check_release_manifest.py does not require it.)

   Test-only simulation is available for local tooling exercises, not release publication:

   python devops/release/produce_image_digests.py \
       --simulate-buildx-fixture-for-tests \
       --manifest stable=devops/releases/2025.09-stable.yaml \
       --manifest airgap=devops/releases/2025.09-airgap.yaml \
       --output out/release/image-digests.test-only.json
   

   The simulated artifact declares testOnly: true and sourceKind: simulated-buildx-fixture. apply_image_digests.py rejects it unless --allow-test-only-artifact-write-for-tests is provided, and any manifest written with that flag is stamped test-only. Production check_release_manifest.py and validate_runtime_posture.py reject those stamped manifests by default; the matching --allow-...-for-tests overrides are only for unit tests and disposable local fixtures.

3. Align deployment bundles with the manifest Run the alignment checker (tools/scripts/validate/check-channel-alignment.py) for every profile that should pick up the release. --release, --target, and --ignore-repo (repeatable) are the supported flags. Pass --ignore-repo nats to skip auxiliary services.

   python tools/scripts/validate/check-channel-alignment.py \
       --release devops/releases/2025.10-edge.yaml \
       --target devops/compose/docker-compose.stella-ops.yml \
       --ignore-repo nats
   

   Repeat for other channels (stable, airgap), substituting the manifest and target files.

   Note: the per-channel docker-compose.{dev,stage,prod,airgap}.yaml targets documented historically do not exist (see §1). Until they are materialized, point --target at a profile that actually exists (docker-compose.stella-ops.yml, or the infra-only docker-compose.dev.yml).

4. Lint and template profiles

   ./tools/scripts/validate/validate-profiles.sh
   

   The script validates the current compose files and fails if active Helm charts are reintroduced. If Docker/Compose is unavailable, it reports the missing local tooling and skips compose rendering rather than inventing profiles that do not exist.

5. Smoke the Offline Kit debug store (edge/stable only) When the release pipeline has generated out/release/debug/.build-id/**, mirror the assets into the Offline Kit staging tree. The helper accepts --release-dir, --offline-kit-dir, --verify-only, and --dry-run:

   ./devops/offline/kit/mirror_debug_store.py \
       --release-dir out/release \
       --offline-kit-dir out/offline-kit
   

   Archive the resulting out/offline-kit/metadata/debug-store.json alongside the kit bundle.

6. Review compatibility matrix Confirm PostgreSQL, Valkey, and RustFS images in the release manifest match platform SLOs and are digest-pinned. Version tags such as rustfs:2025.09.2 are identifiers only; they are not sufficient release evidence without @sha256. If the Sigstore tools profile is enabled, verify rekor-cli:v1.4.3 and cosign:v3.0.4.

7. Create a rollback bookmark Record the current compose tag (git describe --tags) before applying changes.

3. Kubernetes / Helm: not supported

There is no Helm upgrade path. Stella Ops does not support Kubernetes or Helm deployments, and devops/helm/stellaops has been removed from active packaging. Use the Docker Compose upgrade procedure below.

4. Docker Compose upgrade procedure

Customer installations run the release bundle, not this compose tree. The steps in this section apply to a checkout of this repository. A release-bundle installation upgrades in place with the newer bundle’s installer: see Release-bundle installations.

Profile paths below are illustrative. A dedicated docker-compose.prod.yaml profile and a devops/compose/env/prod.env file do not exist in the current tree (see §1). The full platform stack ships as devops/compose/docker-compose.stella-ops.yml with environment defaults in devops/compose/env/stellaops.env.example. Substitute your site-specific env file (copied from the relevant *.env.example) for --env-file.

Router cancellation rollout gate (Q-8; executed 2026-09-01, SPRINT_20260824_010 P8-15). The signed per-connection control lane is the only cancellation path. Upgrade every messaging microservice first and verify that each replica advertises messaging.cancel-control.v1; only then upgrade the gateway. A non-advertising replica receives no cancellations; treat a fleet-wide missing capability as a deploy-order defect, not a reason to restore dual-delivery.

1. Update environment files (devops/compose/env/*.env.example, e.g. stellaops.env.example) with any new settings and sync secrets to hosts.
2. Pull the tagged repository state corresponding to the release (e.g. git checkout 2025.09.2 for stable).
3. Apply the upgrade:

   docker compose \
     --env-file devops/compose/env/<site>.env \
     -f devops/compose/docker-compose.stella-ops.yml \
     pull
   
   docker compose \
     --env-file devops/compose/env/<site>.env \
     -f devops/compose/docker-compose.stella-ops.yml \
     up -d
   

4. Verify every service converged its own schema at startup:

   docker compose ps
   docker compose logs --since 5m | grep 'Migration:'
   

   Expected result: no container restarting; each service logs Migration: Database is up to date for <Module>. or the migrations it applied.
5. Stop condition: a service exits with Pending manual-category migrations block startup or Checksum mismatch. Follow docs/runbooks/database/migration-recovery.md (renumber a Release/Data-band file into the Startup band so the owning host applies it, or apply it by hand and record the ledger row; checksum drift has its own procedure there).
6. Tail logs for critical services (docker compose logs -f authority concelier).
7. Update monitoring dashboards/alerts to confirm normal operation.

Migration notes:

• Compose PostgreSQL init scripts in devops/compose/postgres-init are first-initialization only; 00-v1-baseline.sql creates only stellaops_authority and Authority’s own shells, nothing any service owns.
• Every service converges its own database from its own embedded Startup-band SQL at host start (AddStartupMigrations); one service = one database + own role (ADR-039). Consult docs/db/MIGRATION_INVENTORY.md before production upgrades.
• Consolidation target policy and cutover waves are documented in docs/db/MIGRATION_CONSOLIDATION_PLAN.md.
• Release-category (100+ or rollback in the name) and Data-category (DM-prefixed) migrations must not exist; NoManualCategoryMigrationsTests guards the tree, and the owning host fails closed if one appears.
• Retired (SPRINT_20260722_021 PLT-4, 2026-09-14): the stella system migrations-status | migrations-verify | migrations-run sequence this runbook used to prescribe as the release migration gate was deleted with the central migration-plugin mechanism. There is no manual or DR migration applier.
• There is no UI-driven migration execution, and the API it would have called is gone. The Platform admin APIs (/api/v1/admin/migrations/*) were retired by SPRINT_20260722_026 CM-2 on 2026-08-02, endpoints, gateway route and all; the Console never shipped a page that called them. Migration operations are per-service: convergence on service startup (§2.7) and status through each service’s db.migration-status doctor check (GET /doctor/<service>/checks). A browser must still never connect directly to PostgreSQL.

Rollback (Compose)

1. Check out the previous release tag (e.g. git checkout 2025.09.1).
2. Capture migration state before rollback (per owning database and schema):

   docker compose exec postgres psql -U stellaops -d <db> \
     -c "select migration_name, category, applied_at from <schema>.schema_migrations order by applied_at desc limit 10;"
   

3. Re-run docker compose pull and docker compose up -d with that profile. Docker will restore the prior digests.
4. If schema incompatibility is detected, restore known-good database snapshots before service restart (see docs/modules/authority/operations/backup-restore.md and associated service guides). Migrations are forward-only (ADR-004), so the snapshot to restore is the complete backup taken before the upgrade (backup-restore-drill.md, sections 4 and 6).
5. Re-check migration status and log rollback actions in the operations journal.

Release-bundle installations

Verified against source at cdc04578c2 on 2026-10-12 (SPRINT_20261007_029 CCP-065). The procedure is tested on its source boundaries (see “Re-verify” below). It has not been run on a real installation yet; the CA-14 acceptance drill (CCP-053) records the first run.

A customer installation runs the release bundle (devops/release/bundle/). It upgrades in place: the newer bundle is extracted over the installation directory, and the newer bundle’s installer is run there. The installation keeps its directory, its .env, its generated keys, its Docker names and its data volumes; the bundle files, the images and the database schemas move to the newer release.

What the newer installer does on an existing installation, in this order:

1. Verifies the newer bundle (signature when present, every checksum in its release-manifest.yaml).
2. Removes the older bundle’s files that the newer bundle no longer ships. state/installed-release-manifest.yaml records the manifest of the bundle whose files the directory holds. The installer removes each file listed there and not in the newer manifest, only when its bytes still match the recorded checksum and it lies inside the installation directory; a changed file is kept and named. It then records the newer manifest. An installation made with a bundle that predates this record has none: step 3 of the procedure creates it.
3. Reconciles .env. Every existing value is kept. Each setting the newer .env.example declares and .env lacks (a commented-out line counts as absent) is appended, drawn by the same rule a fresh install uses: a separate random value per GENERATED_* placeholder, each service database password inside its STELLAOPS_POSTGRES_<SERVICE>_CONNECTION, the shipped value otherwise; STELLAOPS_DEPLOYMENT_BUNDLE_KEK and NOTIFY_DOCTOR_CLIENT_SECRET copy their source setting. The installer prints the name of each added setting, never a value, and names each kept setting whose value differs from the newer bundle’s shipped default. It refuses, and leaves .env unchanged, when .env lacks POSTGRES_PASSWORD or STELLAOPS_ADMIN_PASS: those belong to the existing cluster and administrator and cannot be drawn again.
4. Keeps every generated key and certificate, and checks the Docker names as on any re-run.
5. Pulls the newer images (unless --offline / -Offline).
6. Provisions the service databases the newer bundle adds. It starts postgres alone, waits until it accepts TCP connections, and runs config/postgres-init/20-service-databases.sh --existing-cluster inside it. A service whose database is missing gets its role (with the password from its connection setting), its database and its CONNECT isolation, exactly as on a first start; the cluster-level NOLOGIN roles the newer bundle needs are created when missing. A service whose database exists keeps its role, password and data. A database that exists under another owner stops the run before anything is created. No other service starts until this step succeeded.
7. Starts the stack. Each service migrates its own database forward at startup (ADR-004); the installer waits for health and imports a newer vulnerability-data seed when the bundle carries one.

Steps 2, 3 and 6 run on every full start, so a re-run of the same bundle changes nothing: same manifest, every setting present, every database kept.

Before you start.

• Take a complete backup of the installation (backup-restore-drill.md, section 4). It is the only rollback: the newer services migrate the existing databases forward at their first start (ADR-004), and running the older bundle against migrated databases is not supported.
• Verify the newer archive (and its separately published mirror-seed.v1.ndjson, when its SHA256SUMS lists one) against SHA256SUMS. Keep the seed in the directory that contains the installation directory: the installer looks for it there.
• This procedure keeps the PostgreSQL data directory. It does not cover a newer bundle whose POSTGRES_IMAGE is another PostgreSQL major version; that needs a dump and restore, which the bundle does not provide.

Procedure. Run in the installation directory; compose stands for docker compose --env-file .env -f docker-compose.pinned.yml -f secrets/docker-compose.installation.yml, plus -f docker-compose.demo-registry.yml for an installation made with --with-demo-registry (-WithDemoRegistry).

1. Back up (section 4 of the drill), then stop the installation and keep its volumes:

   compose down          # never down -v
   

2. Note the installer options the installation was made with (--with-demo-registry, --offline); the newer installer needs the same ones.

3. Only when state/installed-release-manifest.yaml does not exist (an installation made before this record), record the older bundle’s files before the newer bundle overwrites its manifest:

   mkdir -p state && cp -n release-manifest.yaml state/installed-release-manifest.yaml
   

   New-Item -ItemType Directory -Force state | Out-Null
   if (-not (Test-Path state\installed-release-manifest.yaml)) { Copy-Item release-manifest.yaml state\installed-release-manifest.yaml }
   

   Without the record nothing is removed in step 2 of the installer, and a file the older bundle shipped and the newer one does not stays in the directory.

4. Extract the newer bundle over the installation directory. The archive holds one top-level directory; drop it:

   tar -xzpf ../stellaops-bundle-<newer>.tar.gz --strip-components=1 -C .
   

   tar -xzf ..\stellaops-bundle-<newer>.tar.gz --strip-components=1 -C .
   

5. Run the newer installer with the same options: ./install.sh (.\install.ps1).

6. Read its output before you use the installation:

   • added <NAME>: a setting the newer bundle introduced. A generated secret needs nothing. An installation input the newer bundle leaves empty (as a fresh install would, for example a new trust path) must be filled in .env; the newer stack refuses to start on an empty required value and names it. Fill it and run the installer again.
   • kept <NAME> (differs from this bundle's default; review it): .env keeps its value. For a value the release owns, set the newer bundle’s value unless the installation chose its own on purpose: the infrastructure image pins POSTGRES_IMAGE, VALKEY_IMAGE, RUSTFS_IMAGE and ZOT_IMAGE (release identity, release-manifest.yaml infrastructure:; keep a mirrored image, never another PostgreSQL major version) and settings such as SIGNALS_ASPNETCORE_ENV. Run the installer again after a change.
   • removed <path> / kept <path>: ... its bytes changed since: older bundle files. Move a kept file aside when nothing of the installation uses it.
   • bundle database init: provisioned stellaops_<service> for <service> / kept stellaops_<service> for <service>: the database step.

7. Verify:

   compose ps                                          # every service running, healthy where it has a check
   compose logs --since 30m | grep 'Migration:'        # each service converged its own schema
   compose exec -T postgres psql -U stellaops -d postgres -tAc \
     "select datname || ' ' || pg_get_userbyid(datdba) from pg_database where datname like 'stellaops\_%' order by 1"
   cmp release-manifest.yaml state/installed-release-manifest.yaml   # the directory holds the newer bundle
   

   Every service database is owned by its own role (stellaops_<service> <service>). Compare the image every service runs with the newer manifest as in section 8 (/buildinfo.json).

Stop conditions.

• .env has no POSTGRES_PASSWORD (or STELLAOPS_ADMIN_PASS): .env is not this installation’s. .env was not changed and no service was started; the installer’s step 2 (the older bundle’s files) had already run. Restore .env from the backup and run the installer again.
• bundle database init: stellaops_<x> ... exists and is owned by <y>, not by <x>: the cluster does not match .env. Nothing was changed; restore the installation directory and the cluster from the same backup.
• postgres did not accept connections: read compose logs postgres (a PostgreSQL major version change shows here). No other service was started.
• A service stops with Pending manual-category migrations block startup or Checksum mismatch: migration-recovery.md.
• To roll back, restore the complete backup taken in step 1 with the older bundle’s files (backup-restore-drill.md, section 6).

Known limits.

• A kept value is never changed. The installer cannot tell a value the older bundle shipped from one the operator chose, because the extraction replaces the older .env.example; it names the differing ones instead. An empty value is kept empty as well.
• The installer does not migrate a PostgreSQL major version or move an installation between hosts.

Example, the bundle pair named for the CA-14 drill: bundle 17cb878837 adds 20 settings to .env.example over e61705dca1. Five of them are required by its compose file (${…:?}): the database connection STELLAOPS_POSTGRES_FINDINGS_CONNECTION and the generated secrets FINDINGS_ATTACHMENTS_CSRF_SHARED_SECRET, FINDINGS_ATTACHMENTS_ENCRYPTION_KEY, FINDINGS_ATTACHMENTS_SIGNED_URL_SECRET and SCANNER_SBOM_EXPORT_RECEIPT_HMAC_SECRET_B64. The steps above belong to the NEWER bundle’s installer, and bundle 17cb878837 predates them: its installer keeps .env as it is and creates no database on an existing cluster, so that pair upgrades only to a bundle built from source that carries these steps. The tests replay the pair’s change with this installer: it adds all 20 settings, the five required ones drawn, provisions the findings database and role and the findings_ledger_admin NOLOGIN role on a cluster that bundle e61705dca1’s own init files created, and names SIGNALS_ASPNETCORE_ENV and JOBENGINE_ASPNETCORE_ENV, which the older installation keeps at Development where the newer bundle ships Production.

Re-verify:

git diff e61705dca1 17cb878837 -- devops/release/bundle/.env.example
# .env reconcile of both installers on that pair, on e61705dca1 to this tree, and refusals
python3 devops/release/bundle/tools/test_install_bundle_upgrade.py
# --existing-cluster in a disposable container of the bundle's PostgreSQL image (needs Docker)
python3 devops/release/bundle/tools/test_bundle_database_upgrade.py
bash devops/release/bundle/install.sh --self-test     # includes the reconcile, prune and provisioning order
pwsh devops/release/bundle/install.ps1 -SelfTest      # Windows sibling

5. Channel promotion workflow

1. Author or update the channel manifest under devops/releases/.
2. Mirror the new digests into Compose profiles and Offline Kit metadata and run the alignment script for each supported profile. (Until the per-channel Compose profiles exist — see §1 — the “mirror into Compose profiles” step has no concrete target; mirror into the manifests + Offline Kit metadata only, and run the alignment script against docker-compose.stella-ops.yml.)
3. Commit the changes with a message that references the release version and channel (e.g. deploy: promote 2025.10.0-edge).
4. Publish release notes and update devops/releases/README.md (note: this README does not currently exist in devops/releases/ — create it if release engineering adopts a per-channel changelog).
5. Tag the repository when promoting stable or airgap builds.

6. Upgrade rehearsal & rollback drill log

Maintain rehearsal notes in docs/runbooks/deploy/launch-cutover.md or the relevant sprint planning document. After each drill capture:

• Release version tested
• Date/time
• Participants
• Issues encountered & fixes
• Rollback duration (if executed)

Attach the log to the sprint retro or operational wiki.

7. References

• docs/operations/devops/architecture.md– DevOps release & operations architecture, SLOs, and compliance requirements.
• docs/operations/deployment/docker.md– Docker & Compose deployment guide.
• docs/releases/RELEASE_ENGINEERING_PLAYBOOK.md – release automation and signing pipeline.
• devops/README.md – structure and validation workflow for deployment bundles.
• devops/offline/kit/mirror_debug_store.py – debug-store mirroring helper.
• tools/scripts/validate/check-channel-alignment.py – release vs. deployment digest alignment checker.

8. Image-staleness self-check (/buildinfo.json)

Added 2026-05-17 (Sprint 20260517_025, OPS-015). Supersedes the OPS-007 / OPS-041 docker inspect + git log --since workflow.

Every backend service image baked via devops/docker/build-service-publish.sh (Sprint 20260505_008) carries /app/buildinfo.json with the source provenance of the publish step. The file is served unauthenticated by each service at GET /buildinfo.json (and the /api/v1/buildinfo alias) so operators can probe drift without exec’ing into the container.

Canonical operator probe

# Service-direct (forwarded by the gateway to the service):
curl -sk https://stella-ops.local/buildinfo.json | jq .

# Expected payload:
# {
#   "module": "platform",
#   "version": "dev",
#   "gitSha": "8282a08e74...",
#   "gitCommitTime": "2026-05-17T09:37:57+03:00",
#   "imageBuiltAt": "2026-05-17T06:51:31Z",
#   "branch": "main"
# }

A wider drift sweep iterates the known service hostnames behind the gateway (each backend mounts the same handler):

for svc in platform notify-web scanner-web findings-ledger-web router-gateway concelier; do
  echo "== ${svc} =="
  docker exec "stellaops-${svc}" cat /app/buildinfo.json 2>/dev/null || echo '  (no buildinfo — image predates Sprint 025)'
done

What to compare

The image is stale if imageBuiltAt < the commit time of any change under the service’s src/<Module>/ subtree on the source branch. Compute with:

git -C <repo> log --since="$(jq -r .imageBuiltAt /tmp/buildinfo.json)" -- src/<Module>

For operator evidence, prefer the consolidated read-only drift report instead of manually combining Docker and git output:

powershell -NoProfile -ExecutionPolicy Bypass -File tools/scripts/qa/Invoke-ImageSourceDriftReport.ps1 -OutputDirectory artifacts/qa/image-source-drift/latest

The command writes JSON and Markdown reports containing the Docker image state, service buildinfo, mapped source path, post-image commit count, and per-service verdict. It exits 0 for PASS/PARTIAL, 1 for DRIFT, and 2 for a blocked live input such as an unavailable Docker engine.

Fallback behaviour

If /app/buildinfo.json is missing (image was built before this pattern landed), the handler still responds 200 with gitSha=unbaked and a 1970 timestamp, plus a startup warning in the service log:

Build-info file missing at /app/buildinfo.json; serving synthesized placeholder for module <name>.

Rebuild via devops/docker/build-service-publish.sh <service-key> to bake provenance into the next image.

Gateway aggregator (implemented)

Implemented in Sprint 20260517_031 (OPS-015 follow-up F3); supersedes the earlier “planned” note.

The gateway now exposes a fan-out aggregator at GET /buildinfo.json (gateway root) that returns one buildinfo entry per HELLO-registered service. It is anonymous by design — staleness is a public operational signal that must be visible before login so operators can detect image drift in degraded/auth-broken environments. The response carries a Cache-Control: public, max-age=30 header mirroring the in-process cache TTL.

# Cluster-wide drift sweep via the gateway aggregator:
curl -sk https://stella-ops.local/buildinfo.json | jq .

Source: src/Router/StellaOps.Gateway.WebService/Endpoints/BuildInfoAggregatorEndpoints.cs (AggregatorPath = "/buildinfo.json"). Note the canonical aggregator path is the gateway root /buildinfo.json, not /api/v1/buildinfo. You can still probe each service directly (the gateway also forwards a per-service /buildinfo.json to the matching backend).