Stella Ops Deployment Upgrade & Rollback Runbook

Audience: release engineers and platform operators promoting or rolling back a Stella Ops release. Scope: promote / roll back a release and keep the edge, stable, and airgap channels aligned.

This runbook describes how to promote a new release (Docker Compose and Offline Kit), how to roll back safely, and how to keep the edge, stable, and airgap channels aligned. All steps assume you are working from a clean checkout of the release branch/tag.

Tree-vs-doc caveat. The per-channel docker-compose.{dev,stage,prod,airgap}.yaml deployment profiles and env/{...}.env files that this runbook (and the release tooling) historically assume do not exist in the current devops/compose/ tree. The release-manifest authority, digest tooling (devops/release/*.py), alignment checker, CLI migration gate, and buildinfo probes are all real and verified; the deployment-profile layer is not. See §1.

Maintenance: update this runbook whenever the release digest inventory, release/evidence schemas, default local-model guidance, or provider-pack guidance changes.

1. Channel overview

(Two additional manifests live alongside these in devops/releases/: 2025.09-mock-dev.yaml and the machine-readable service-versions.json. They are not promotion channels.)

Compose-profile reality check (verified against devops/compose/). The full platform is run from devops/compose/docker-compose.stella-ops.yml. The infrastructure-only development stack remains devops/compose/docker-compose.dev.yml (note .yml). Release tooling and profile validation now point at files that actually exist under devops/; the historical deploy/ and devops/helm/stellaops targets are obsolete. Dedicated per-channel docker-compose.prod.yaml / stage / airgap files still do not exist, so channel promotion uses the release manifest plus the current compose stack or site-specific operator overlays.

Infrastructure components must be digest-pinned in the release manifests. PostgreSQL and Valkey are digest-pinned in the 2025.09 stable and air-gap manifests; RustFS is still tag-only (registry.stella-ops.org/stellaops/rustfs:2025.09.2) there and blocks strict production posture until release engineering publishes an authoritative digest.

Manifest digest gaps (verified against devops/releases/*.yaml, 2026-05-30). The infrastructure caveat above is not the whole story — several component images are not digest-pinned either, and one required component is missing outright:

• advisory-ai-web and advisory-ai-worker are tag-only in all three manifests (:2025.09.2, :2025.09.2-airgap, :2025.10.0-edge).
• notify-web is a required component per devops/release/check_release_manifest.py but is absent from the stable, air-gap, and edge manifests entirely, so the validator fails with missing required component 'notify-web'.
• The edge manifest additionally carries issuer-directory-web tag-only.

Consequently check_release_manifest.py does not pass against the manifests as checked in; the digest-pinned entries (Authority, Signer, Attestor, Scanner Web/Worker, Concelier, Excititor, Web UI, plus PostgreSQL and Valkey) are real, but the manifests are not yet release-publishable.

2. Pre-flight checklist

1. Refresh release manifest Pull the latest manifest for the channel you are promoting (devops/releases/<version>-<channel>.yaml).

2. Validate release authority before publication Stable and air-gap release manifests must carry the digest-pinned authority for every production image before values files are updated. This check is local-only; it does not resolve mutable tags or query registries.

   Produce the local digest artifact from build/push outputs first. Component digests must come from Docker Buildx --metadata-file outputs written by successful publish jobs; one <component>.metadata.json file is required for every component in devops/release/check_release_manifest.py. The producer can carry already digest-pinned PostgreSQL and Valkey entries from the local release manifests, but RustFS is tag-only in the current 2025.09 manifests and must be supplied by an infrastructure source captured from a real local mirror/build result.

   python devops/release/produce_image_digests.py \
       --channel-source stable=out/release/stable/artifacts/metadata \
       --channel-source airgap=out/release/airgap/artifacts/metadata \
       --manifest stable=devops/releases/2025.09-stable.yaml \
       --manifest airgap=devops/releases/2025.09-airgap.yaml \
       --infrastructure-source out/release/infrastructure-digests.json \
       --output out/release/image-digests.json
   

   The infrastructure source is local JSON with manifests.<channel>.infrastructure mappings to final image@sha256:<digest> references. It is required for any infrastructure image that is not already digest-pinned in the local release manifest. Tags, missing entries, repository/name mismatches, missing Buildx digests, and one-channel-only updates fail before any manifest is written.

   Apply the produced artifact only after the producer succeeds:

   python devops/release/apply_image_digests.py \
       --artifact out/release/image-digests.json \
       --manifest devops/releases/2025.09-stable.yaml \
       --manifest devops/releases/2025.09-airgap.yaml
   

   Artifact shape:

   {
     "manifests": {
       "stable": {
         "components": {
           "authority": "registry.stella-ops.org/stellaops/authority@sha256:<64-hex>",
           "notify-web": "registry.stella-ops.org/stellaops/notify-web@sha256:<64-hex>",
           "advisory-ai-web": "registry.stella-ops.org/stellaops/advisory-ai-web@sha256:<64-hex>",
           "advisory-ai-worker": "registry.stella-ops.org/stellaops/advisory-ai-worker@sha256:<64-hex>"
         },
         "infrastructure": {
           "postgres": "docker.io/library/postgres@sha256:<64-hex>",
           "valkey": "docker.io/valkey/valkey@sha256:<64-hex>",
           "rustfs": "registry.stella-ops.org/stellaops/rustfs@sha256:<64-hex>"
         }
       },
       "airgap": {
         "components": {
           "authority": "registry.stella-ops.org/stellaops/authority@sha256:<64-hex>",
           "notify-web": "registry.stella-ops.org/stellaops/notify-web@sha256:<64-hex>",
           "advisory-ai-web": "registry.stella-ops.org/stellaops/advisory-ai-web@sha256:<64-hex>",
           "advisory-ai-worker": "registry.stella-ops.org/stellaops/advisory-ai-worker@sha256:<64-hex>"
         },
         "infrastructure": {
           "postgres": "docker.io/library/postgres@sha256:<64-hex>",
           "valkey": "docker.io/valkey/valkey@sha256:<64-hex>",
           "rustfs": "registry.stella-ops.org/stellaops/rustfs@sha256:<64-hex>"
         }
       }
     }
   }
   

   The example abbreviates the component list for readability. The real artifact must include every component required by devops/release/check_release_manifest.py: Authority, Signer, Attestor, Scanner Web/Worker, Concelier, Excititor, Notify Web, AdvisoryAI Web/Worker, and Web UI.

   python devops/release/check_release_manifest.py \
       --manifest devops/releases/2025.09-stable.yaml \
       --manifest devops/releases/2025.09-airgap.yaml
   

   If this fails, release engineering must add the missing component metadata or replace the tag-only entry with the attested image@sha256:<digest> from the signed release build output. Current blockers (verified 2026-05-30 against the checked-in manifests): notify-web is missing entirely as a component (the validator reports missing required component 'notify-web'); advisory-ai-web and advisory-ai-worker are present but tag-only; and rustfs is tag-only in infrastructure. RustFS specifically has no local authoritative source in the current tree beyond tag-only manifest entries. (The edge manifest also carries issuer-directory-web tag-only, but check_release_manifest.py does not require it.)

   Test-only simulation is available for local tooling exercises, not release publication:

   python devops/release/produce_image_digests.py \
       --simulate-buildx-fixture-for-tests \
       --manifest stable=devops/releases/2025.09-stable.yaml \
       --manifest airgap=devops/releases/2025.09-airgap.yaml \
       --output out/release/image-digests.test-only.json
   

   The simulated artifact declares testOnly: true and sourceKind: simulated-buildx-fixture. apply_image_digests.py rejects it unless --allow-test-only-artifact-write-for-tests is provided, and any manifest written with that flag is stamped test-only. Production check_release_manifest.py and validate_runtime_posture.py reject those stamped manifests by default; the matching --allow-...-for-tests overrides are only for unit tests and disposable local fixtures.

3. Align deployment bundles with the manifest Run the alignment checker (devops/tools/check-channel-alignment.py) for every profile that should pick up the release. --release, --target, and --ignore-repo (repeatable) are the supported flags. Pass --ignore-repo nats to skip auxiliary services.

   python devops/tools/check-channel-alignment.py \
       --release devops/releases/2025.10-edge.yaml \
       --target devops/compose/docker-compose.stella-ops.yml \
       --ignore-repo nats
   

   Repeat for other channels (stable, airgap), substituting the manifest and target files.

   Note: the per-channel docker-compose.{dev,stage,prod,airgap}.yaml targets documented historically do not exist (see §1). Until they are materialized, point --target at a profile that actually exists (docker-compose.stella-ops.yml, or the infra-only docker-compose.dev.yml).

4. Lint and template profiles

   ./devops/tools/validate-profiles.sh
   

   The script validates the current compose files and fails if active Helm charts are reintroduced. If Docker/Compose is unavailable, it reports the missing local tooling and skips compose rendering rather than inventing profiles that do not exist.

5. Smoke the Offline Kit debug store (edge/stable only) When the release pipeline has generated out/release/debug/.build-id/**, mirror the assets into the Offline Kit staging tree. The helper accepts --release-dir, --offline-kit-dir, --verify-only, and --dry-run:

   ./devops/offline/kit/mirror_debug_store.py \
       --release-dir out/release \
       --offline-kit-dir out/offline-kit
   

   Archive the resulting out/offline-kit/metadata/debug-store.json alongside the kit bundle.

6. Review compatibility matrix Confirm PostgreSQL, Valkey, and RustFS images in the release manifest match platform SLOs and are digest-pinned. Version tags such as rustfs:2025.09.2 are identifiers only; they are not sufficient release evidence without @sha256. If the Sigstore tools profile is enabled, verify rekor-cli:v1.4.3 and cosign:v3.0.4.

7. Create a rollback bookmark Record the current compose tag (git describe --tags) before applying changes.

3. Kubernetes / Helm: not supported

There is no Helm upgrade path. Stella Ops does not support Kubernetes or Helm deployments, and devops/helm/stellaops has been removed from active packaging. Use the Docker Compose upgrade procedure below.

4. Docker Compose upgrade procedure

Profile paths below are illustrative. A dedicated docker-compose.prod.yaml profile and a devops/compose/env/prod.env file do not exist in the current tree (see §1). The full platform stack ships as devops/compose/docker-compose.stella-ops.yml with environment defaults in devops/compose/env/stellaops.env.example. Substitute your site-specific env file (copied from the relevant *.env.example) for --env-file.

1. Update environment files (devops/compose/env/*.env.example, e.g. stellaops.env.example) with any new settings and sync secrets to hosts.
2. Pull the tagged repository state corresponding to the release (e.g. git checkout 2025.09.2 for stable).
3. Apply the upgrade:

   docker compose \
     --env-file devops/compose/env/<site>.env \
     -f devops/compose/docker-compose.stella-ops.yml \
     pull
   
   docker compose \
     --env-file devops/compose/env/<site>.env \
     -f devops/compose/docker-compose.stella-ops.yml \
     up -d
   

4. Run migration status and verification checks:

   stella system migrations-status --module all
   stella system migrations-verify --module all
   stella system migrations-run --module all --category release --dry-run
   

5. Execute release migrations when approved:

   stella system migrations-run --module all --category release --force
   stella system migrations-status --module all
   

6. Tail logs for critical services (docker compose logs -f authority concelier).
7. Update monitoring dashboards/alerts to confirm normal operation.

Migration notes:

• Compose PostgreSQL init scripts in devops/compose/postgres-init are first-initialization only.
• CLI module coverage is currently limited; consult docs/db/MIGRATION_INVENTORY.md before production upgrades.
• Consolidation target policy and cutover waves are documented in docs/db/MIGRATION_CONSOLIDATION_PLAN.md.
• On empty migration history, CLI/API paths synthesize one per-service consolidated migration (100_consolidated_<service>.sql) and backfill legacy migration history rows for future update compatibility.
• If consolidated history exists with partial legacy backfill, CLI/API paths auto-backfill missing legacy rows before source-set execution.
• For upgradeable on-prem installations, the stella system migrations-* sequence is the required release migration gate.
• UI-driven migration execution must call Platform admin APIs (/api/v1/admin/migrations/*) and not connect directly from browser to PostgreSQL.

Rollback (Compose)

1. Check out the previous release tag (e.g. git checkout 2025.09.1).
2. Capture migration state before rollback:

   stella system migrations-status --module all
   

3. Re-run docker compose pull and docker compose up -d with that profile. Docker will restore the prior digests.
4. If schema incompatibility is detected, restore known-good database snapshots before service restart (see docs/modules/authority/operations/backup-restore.md and associated service guides).
5. Re-check migration status and log rollback actions in the operations journal.

5. Channel promotion workflow

1. Author or update the channel manifest under devops/releases/.
2. Mirror the new digests into Compose profiles and Offline Kit metadata and run the alignment script for each supported profile. (Until the per-channel Compose profiles exist — see §1 — the “mirror into Compose profiles” step has no concrete target; mirror into the manifests + Offline Kit metadata only, and run the alignment script against docker-compose.stella-ops.yml.)
3. Commit the changes with a message that references the release version and channel (e.g. deploy: promote 2025.10.0-edge).
4. Publish release notes and update devops/releases/README.md (note: this README does not currently exist in devops/releases/ — create it if release engineering adopts a per-channel changelog).
5. Tag the repository when promoting stable or airgap builds.

6. Upgrade rehearsal & rollback drill log

Maintain rehearsal notes in docs/operations/devops/runbooks/launch-cutover.md or the relevant sprint planning document. After each drill capture:

• Release version tested
• Date/time
• Participants
• Issues encountered & fixes
• Rollback duration (if executed)

Attach the log to the sprint retro or operational wiki.

7. References

• docs/operations/devops/architecture.md– DevOps release & operations architecture, SLOs, and compliance requirements.
• docs/operations/deployment/docker.md– Docker & Compose deployment guide.
• docs/releases/RELEASE_ENGINEERING_PLAYBOOK.md– release automation and signing pipeline.
• devops/README.md – structure and validation workflow for deployment bundles.
• devops/offline/kit/mirror_debug_store.py – debug-store mirroring helper.
• devops/tools/check-channel-alignment.py – release vs. deployment digest alignment checker.

8. Image-staleness self-check (/buildinfo.json)

Added 2026-05-17 (Sprint 20260517_025, OPS-015). Supersedes the OPS-007 / OPS-041 docker inspect + git log --since workflow.

Every backend service image baked via devops/docker/build-service-publish.sh (Sprint 20260505_008) carries /app/buildinfo.json with the source provenance of the publish step. The file is served unauthenticated by each service at GET /buildinfo.json (and the /api/v1/buildinfo alias) so operators can probe drift without exec’ing into the container.

Canonical operator probe

# Service-direct (forwarded by the gateway to the service):
curl -sk https://stella-ops.local/buildinfo.json | jq .

# Expected payload:
# {
#   "module": "platform",
#   "version": "dev",
#   "gitSha": "8282a08e74...",
#   "gitCommitTime": "2026-05-17T09:37:57+03:00",
#   "imageBuiltAt": "2026-05-17T06:51:31Z",
#   "branch": "main"
# }

A wider drift sweep iterates the known service hostnames behind the gateway (each backend mounts the same handler):

for svc in platform notify-web scanner-web findings-ledger-web router-gateway concelier; do
  echo "== ${svc} =="
  docker exec "stellaops-${svc}" cat /app/buildinfo.json 2>/dev/null || echo '  (no buildinfo — image predates Sprint 025)'
done

What to compare

The image is stale if imageBuiltAt < the commit time of any change under the service’s src/<Module>/ subtree on the source branch. Compute with:

git -C <repo> log --since="$(jq -r .imageBuiltAt /tmp/buildinfo.json)" -- src/<Module>

For operator evidence, prefer the consolidated read-only drift report instead of manually combining Docker and git output:

powershell -NoProfile -ExecutionPolicy Bypass -File tools/scripts/qa/Invoke-ImageSourceDriftReport.ps1 -OutputDirectory artifacts/qa/image-source-drift/latest

The command writes JSON and Markdown reports containing the Docker image state, service buildinfo, mapped source path, post-image commit count, and per-service verdict. It exits 0 for PASS/PARTIAL, 1 for DRIFT, and 2 for a blocked live input such as an unavailable Docker engine.

Fallback behaviour

If /app/buildinfo.json is missing (image was built before this pattern landed), the handler still responds 200 with gitSha=unbaked and a 1970 timestamp, plus a startup warning in the service log:

Build-info file missing at /app/buildinfo.json; serving synthesized placeholder for module <name>.

Rebuild via devops/docker/build-service-publish.sh <service-key> to bake provenance into the next image.

Gateway aggregator (implemented)

Implemented in Sprint 20260517_031 (OPS-015 follow-up F3); supersedes the earlier “planned” note.

The gateway now exposes a fan-out aggregator at GET /buildinfo.json (gateway root) that returns one buildinfo entry per HELLO-registered service. It is anonymous by design — staleness is a public operational signal that must be visible before login so operators can detect image drift in degraded/auth-broken environments. The response carries a Cache-Control: public, max-age=30 header mirroring the in-process cache TTL.

# Cluster-wide drift sweep via the gateway aggregator:
curl -sk https://stella-ops.local/buildinfo.json | jq .

Source: src/Router/StellaOps.Gateway.WebService/Endpoints/BuildInfoAggregatorEndpoints.cs (AggregatorPath = "/buildinfo.json"). Note the canonical aggregator path is the gateway root /buildinfo.json, not /api/v1/buildinfo. You can still probe each service directly (the gateway also forwards a per-service /buildinfo.json to the matching backend).