Launch Cutover Runbook — Stella Ops

Audience: DevOps Guild, module guild reps, and the cutover team running a full-platform launch. Purpose: Drive a staged staging-to-production cutover of the Stella Ops release control plane, with rehearsal, smoke tests, rollback, and approvals.

Document owner: DevOps Guild (2025-10-26). Scope: full-platform launch from staging to production for release 2025.09.2.

Historical record. This runbook captures the original launch cutover. The procedure and command samples below reflect the platform’s state at that time and have not been re-run against the current stack. Two material changes have landed since:

• Storage and data engines changed. MongoDB was fully removed (Sprint 4400) in favour of PostgreSQL; MinIO references are now RustFS; Redis references are now Valkey. The mongodump/mc mirror/runtime_events steps below are kept only as historical context.
• Kubernetes and Helm are unsupported (product decision, 2026-05-01). Active cutovers must use Docker Compose, the Offline Kit, and signed release manifests for non-Kubernetes container estates.

For an active cutover, follow the current Deployment & Upgrade runbook and the up-to-date deployment profiles under deploy/.

1. Roles and Communication

Set up a bridge call 30 minutes before start and keep #launch-bridge updated every 10 minutes.

2. Timeline Overview (UTC)

3. Rehearsal (Staging) Checklist

1. docker network create stellaops_frontdoor || true (if not present on staging jump host).
2. Run deploy/tools/validate-profiles.sh and archive output.
3. Apply staging secrets to the Compose environment file/host secret store; do not create Kubernetes secrets.
4. Deploy the staging Compose profile and verify containers with docker compose ps.
5. Verify health endpoints: curl https://authority.stage.../healthz, curl https://scanner.stage.../healthz.
6. Execute smoke CLI: stellaops-cli scan submit --profile staging --sbom samples/sbom/demo.json and confirm report status in UI.
7. Document total wall time and any deviations in the rehearsal log.

Rehearsal must complete without manual interventions before proceeding to production.

4. Production Cutover Steps

4.1 Pre-flight

• Confirm production secrets in the Compose environment/host secret store contain PostgreSQL, Authority, Signer, Attestor, Concelier, Scanner, Excititor, and Notify connection settings.
• Ensure the external reverse proxy network exists: docker network create stellaops_frontdoor || true on each compose host.
• Back up current configuration and data (historical commands — see the storage note at the top of this runbook; on the current stack take a PostgreSQL snapshot and an object-store backup instead):
  • Mongo snapshot: mongodump --uri "$MONGO_BACKUP_URI" --out /backups/launch-$(date -Iseconds).
  • MinIO policy export: mc mirror --overwrite minio/stellaops minio-backup/stellaops-$(date +%Y%m%d%H%M).

4.2 Apply Updates (Compose)

1. On each compose node, pull updated images for release 2025.09.2:

   docker compose --env-file prod.env -f devops/compose/docker-compose.prod.yaml pull
   

2. Deploy changes:

   docker compose --env-file prod.env -f devops/compose/docker-compose.prod.yaml up -d
   

3. Confirm containers healthy via docker compose ps and docker logs <service> --tail 50.

4.3 Unsupported Kubernetes/Helm Updates

Kubernetes and Helm updates are unsupported. Do not run helm upgrade, kubectl, or Kubernetes rollout commands for Stella Ops.

4.4 Configuration Validation

• Verify Authority issuer metadata: curl https://authority.prod.../.well-known/openid-configuration.
• Validate Signer DSSE endpoint: stellaops-cli signer verify --base-url https://signer.prod... --bundle samples/dsse/demo.json.
• Check Scanner queue connectivity: docker exec stellaops-scanner-web dotnet StellaOps.Scanner.WebService.dll health queue (returns success).
• Ensure Notify (legacy) still accessible while Notifier migration pending.

5. Smoke Tests

Log results in the change ticket with timestamps and screenshots where applicable.

6. Rollback Procedure

1. Assess failure scope; if systemic, initiate rollback immediately while preserving logs/artifacts.
2. For Compose:

   docker compose --env-file prod.env -f devops/compose/docker-compose.prod.yaml down
   docker compose --env-file stage.env -f devops/compose/docker-compose.stage.yaml up -d
   

3. Kubernetes/Helm rollback is unsupported; use the Compose rollback profile only.
4. Restore Mongo snapshot if data inconsistency detected: mongorestore --uri "$MONGO_BACKUP_URI" --drop /backups/launch-<timestamp>.
5. Restore MinIO mirror if required: mc mirror minio-backup/stellaops-<timestamp> minio/stellaops.
6. Notify stakeholders of rollback and capture root cause notes in incident ticket.

7. Post-cutover Actions

• Keep heightened monitoring for 4 hours post cutover; track latency, error rates, and queue depth.
• Confirm audit trails: Authority tokens issued, Scanner events recorded, Attestor submissions stored.
• Update the Launch Readiness Record if any new gaps or follow-ups are discovered.
• Schedule retrospective within 48 hours; include DevOps, module guilds, and product owner.

8. Approval Matrix

Retain all approvals and logs for audit. Update this runbook after each execution to record actual timings and lessons learned.

9. Rehearsal Log