Blue/Green Compose Deployment Runbook

Audience: operators upgrading a single Stella Ops service on a Docker Compose install with a rollback window.

Overview

This runbook describes the Stella Ops compose-level blue/green deployment path. It is an operator-controlled procedure for one service at a time, using devops/compose/docker-compose.blue-green.yml plus the Linux/macOS/WSL and PowerShell switch scripts.

The goal is an atomic service switch with a rollback window, not gradual traffic shaping. For gradual, percentage-based rollout, use the weighted canary deployment runbook instead.

Provenance: SPRINT_20260518_056 (blue/green, tasks C1-C5) and Sprint 057 (router-native weighted canary), both completed and archived under docs-archive/implplan/.

Quick Reference

When to Use Blue/Green vs. Weighted Canary

When the right column applies, use the weighted canary deployment runbook and import the weighted canary Grafana dashboard. The dashboard is backed by Router Gateway metrics from Sprint 057 and is the operator view for request split, per-version error rate, latency, and per-instance counts during the bake window.

Do not use this runbook for a breaking schema migration. Stella Ops database migrations are forward-only under ADR-004. If green applies a schema that blue cannot tolerate, use a maintenance-window upgrade and the migration recovery runbook.

In-Place Multi-Replica Canary

Sprint 057 (completed) added the compose substrate for a weighted canary inside the canonical stack. Use this when the operator wants a small canary percentage instead of the blue/green parallel-stack switch above.

For canary-eligible MVP services, container_name: in devops/compose/docker-compose.stella-services.yml accepts ${INSTANCE_SUFFIX:-}. With INSTANCE_SUFFIX unset, the rendered container names remain the canonical names, for example stellaops-evidence-locker-web. With INSTANCE_SUFFIX=-canary, the same service renders as stellaops-evidence-locker-web-canary.

Validate the name before starting any container:

cd devops/compose
INSTANCE_SUFFIX=-canary docker compose -f docker-compose.stella-ops.yml config \
  | grep 'container_name: stellaops-evidence-locker-web-canary'

Then start only when the local stack state is safe for an additional instance:

INSTANCE_SUFFIX=-canary docker compose -f docker-compose.stella-ops.yml \
  up -d --no-deps evidence-locker-web

After the canary is up, follow weighted-canary-deployment.md for the runtime weight API, admin scopes, Grafana observation, promotion, abort, and rollback steps. Import devops/observability/grafana/canary.json to watch per-version request rate, error rate, latency, and per-instance counts during the bake. Do not use this pattern for authority, router-gateway, platform, postgres, valkey, or other infra/singleton services.

Pre-Flight Checklist

1. Confirm Sprint 055 drain behavior is deployed on the current blue container. The scripts refuse switch when they cannot detect the drain primitive. detect_drain_support / Test-DrainSupport probe in three escalating ways: a STELLAOPS__DRAIN* / DrainDeadline / DrainPollInterval env var, then an ASCII grep, then a UTF-16LE pass. The UTF-16LE pass is the one that matters: the marker is compiled into the .NET DLL as a UTF-16LE string (each character followed by a 0x00 byte), so a plain ASCII grep -a can never match it and gives a false negative. Mirror the script’s UTF-16LE probe for a manual check:

   # Env-var signal (fast path the scripts try first):
   docker inspect stellaops-evidence-locker-web --format '{{json .Config.Env}}' \
     | grep -Ei 'STELLAOPS__DRAIN|DrainDeadline|DrainPollInterval'
   
   # UTF-16LE marker scan (strip NUL bytes before grep, like the scripts do):
   docker exec stellaops-evidence-locker-web sh -lc '
     marker="Stella microservice drain quiesce completed"
     for f in $(find /app -type f \( -name "*.dll" -o -name "*.so" \) 2>/dev/null); do
       tr -d "\000" < "$f" 2>/dev/null | grep -a -m1 -q "$marker" && { echo found; break; }
     done'
   

2. Take a PostgreSQL snapshot before starting a green image that may run migrations. Use migration-recovery.md.

3. Confirm the green image exists and follows the local rollback-tag style used by devops/docker/build-service-publish.sh:

   docker image inspect stellaops/evidence-locker-web:dev-20260518-091030
   docker run --rm --entrypoint cat \
     stellaops/evidence-locker-web:dev-20260518-091030 /app/buildinfo.json
   

4. Record the current blue tag or image ID before switching:

   docker inspect stellaops-evidence-locker-web --format '{{.Image}}'
   docker exec stellaops-evidence-locker-web sh -lc \
     'cat /app/buildinfo.json 2>/dev/null || true'
   

5. Validate the compose overlay without starting containers:

   cd devops/compose
   docker compose -f docker-compose.stella-ops.yml \
     -f docker-compose.blue-green.yml config
   

6. Confirm the service is in the Sprint 056 MVP set: evidence-locker-web, evidence-locker-worker, scanner-web, scanner-worker, concelier, excititor-web, excititor-worker, vexhub-web, vexlens-web, policy-engine, attestor, notify-web.

7. Keep authority, router-gateway, platform, postgres, and infra services out of this flow. Their limitations are listed below.

Standard Operating Procedure

The commands below use evidence-locker-web. Substitute another supported service only after reviewing that service owner’s smoke-test checklist.

1. Bring up the green sibling. This starts evidence-locker-web-green from docker-compose.blue-green.yml without touching the blue container. The script translates --green-tag into the per-service GREEN_IMAGE_TAG_<SERVICE> environment variable the overlay substitutes into the green image: tag (here GREEN_IMAGE_TAG_EVIDENCE_LOCKER_WEB, defaulting to dev when unset) and runs docker compose ... up -d --no-deps <service>-green:

   cd devops/compose
   ./scripts/blue-green-switch.sh \
     --service evidence-locker-web \
     --green-tag dev-20260518-091030 \
     --action bring-up
   

   The equivalent manual invocation, without the helper script, is:

   cd devops/compose
   GREEN_IMAGE_TAG_EVIDENCE_LOCKER_WEB=dev-20260518-091030 \
     docker compose -f docker-compose.stella-ops.yml \
     -f docker-compose.blue-green.yml up -d --no-deps evidence-locker-web-green
   

2. Health-check green before any traffic switch. This probes direct buildinfo and gateway buildinfo. Add --expected-sha <gitSha> if the release evidence already has the target SHA:

   ./scripts/blue-green-switch.sh \
     --service evidence-locker-web \
     --green-tag dev-20260518-091030 \
     --expected-sha <target-git-sha> \
     --action health-check
   

3. Run the manual smoke checklist for the service. buildinfo.json is served anonymously by the BuildInfoEndpointExtensions helper in StellaOps.Hosting, so a bare curl works. The functional endpoints are scope-gated: GET /api/v1/evidence/packs requires the evidence:read scope (EvidenceAuditEndpoints uses StellaOpsResourceServerPolicies.EvidenceRead), so it needs a bearer token or it returns 401:

   curl -k https://stella-ops.local/api/v1/evidence/buildinfo.json
   curl -k -H "Authorization: Bearer <evidence:read token>" \
     https://stella-ops.local/api/v1/evidence/packs
   docker logs --tail=100 stellaops-evidence-locker-web-green
   

4. Switch traffic to green. The script attaches green to the production network alias, checks router visibility, then sends SIGTERM to blue with the drain deadline so Sprint 055 can report Draining and send Goodbye.

   ./scripts/blue-green-switch.sh \
     --service evidence-locker-web \
     --green-tag dev-20260518-091030 \
     --action switch \
     --drain-deadline 30
   

5. Observe the bake window. Default: 15 minutes unless the service owner requires longer. Capture at least:

   docker ps --filter name=stellaops-evidence-locker
   docker logs --since 15m stellaops-router-gateway
   curl -k -H "Authorization: Bearer <evidence:read token>" \
     https://stella-ops.local/api/v1/evidence/packs
   

6. Commit the switch by removing stopped blue:

   ./scripts/blue-green-switch.sh \
     --service evidence-locker-web \
     --green-tag dev-20260518-091030 \
     --action tear-down-blue
   

7. Re-converge to a single-stack topology. Update/redeploy the canonical blue image tag using the normal release process, start the canonical service, and remove green after the new blue has passed smoke tests:

   docker compose -f docker-compose.stella-ops.yml up -d --no-deps evidence-locker-web
   docker stop --time 30 stellaops-evidence-locker-web-green
   docker rm -f stellaops-evidence-locker-web-green
   

Rollback Procedure

Rollback is intended for the bake window before tear-down-blue. It restarts blue, verifies router visibility, drains green, stops green, and removes green.

cd devops/compose
./scripts/blue-green-switch.sh \
  --service evidence-locker-web \
  --green-tag dev-20260518-091030 \
  --action rollback \
  --drain-deadline 30

After rollback, verify blue buildinfo and the service smoke endpoint:

docker inspect stellaops-evidence-locker-web --format '{{.Image}}'
curl -k https://stella-ops.local/api/v1/evidence/buildinfo.json
curl -k -H "Authorization: Bearer <evidence:read token>" \
  https://stella-ops.local/api/v1/evidence/packs

If tear-down-blue already ran, rollback is no longer a blue/green rollback. Use the prior dev-YYYYMMDD-HHMMSS tag from the build-service-publish contract or restore the PostgreSQL snapshot with migration-recovery.md when schema state must also move back.

Troubleshooting

Gateway 503 after switch

Symptom: gateway returns 503 no instances available after green appears healthy. This is consistent with the metadata replay race called out in the Tier 8 plan and tracked from feedback_router_metadata_replay_flake.md.

Workaround:

docker compose -f docker-compose.stella-ops.yml restart router-gateway

Then re-run:

curl -k -H "Authorization: Bearer <router:routing:read token>" \
  'https://stella-ops.local/api/v1/router/instances?service=evidencelocker'
curl -k -H "Authorization: Bearer <evidence:read token>" \
  https://stella-ops.local/api/v1/evidence/packs

/api/v1/router/instances is admin-scoped: the gateway requires a token with the router:routing:read scope (RouterInstancesEndpoints reuses the router.routing.read policy, registered with AddStellaOpsScopePolicy in the gateway Program.cs). A request with no/invalid token returns 401; an authenticated token that lacks the scope returns 403 (the policy chains RequireAuthenticatedUser then RequireClaim). The ?service= filter matches the instance’s registered service name, which for these services is the router consumer-group name (evidencelocker, scanner, concelier, policy-engine, …) — the same value the switch scripts query — not the compose service name.

Green fails health-check

Do not switch. Remove green and investigate logs:

docker logs --tail=200 stellaops-evidence-locker-web-green
docker stop stellaops-evidence-locker-web-green
docker rm -f stellaops-evidence-locker-web-green

Switch refuses with “drain primitive not detected”

Blue was built before Sprint 055, the container does not contain the drain strings, or the probe could not inspect the image. Redeploy blue from a Sprint 055+ image first, then run the switch again.

Router instances API is unavailable

The PowerShell script fails the switch if /api/v1/router/instances?service=<name> is not reachable. The bash script also checks recent gateway logs as a secondary signal, but operators should not treat log fallback as final release evidence. Capture the failure and decide whether the Router admin API owner must expose the endpoint.

Note: the switch scripts call this endpoint without an authorization header (both verify_gateway_instances in the bash script and Confirm-GatewayInstances in the PowerShell script issue an unauthenticated curl), so the unauthenticated 401 the admin-scoped endpoint returns also presents as “unavailable.” The endpoint requires a token with router:routing:read. If the script reports the API unavailable but the gateway is healthy, confirm the call by re-running it with a router:routing:read bearer token; if that returns the instance list, the script failure was the no-token 401, not an absent endpoint.

Network alias does not move

The scripts reconnect the target container to the stellaops network with the green and production aliases because Docker cannot add a new alias to an already-connected endpoint. If this fails, stop and inspect the network:

docker network inspect stellaops
docker inspect stellaops-evidence-locker-web-green

Limitations

• Authority is excluded. The token issuer cannot be split-brain in this MVP.
• Router-gateway is excluded. It is the switch mechanism and needs a separate ingress-level blue/green plan.
• Platform is excluded from this overlay because it aggregates control-plane state and participates in setup/admin flows.
• PostgreSQL is not blue/green at the compose layer. ADR-004 defines the platform position: forward-only migrations, snapshot restore for rollback.
• Green startup may apply migrations before the traffic switch. The new schema is shared with blue immediately. Blue must remain forward-compatible for the bake window.
• Worker services can be started as green siblings, but they do not have router consumer groups in the current compose file. Use them for image validation and controlled worker cutover, not gateway traffic splitting.
• The scripts do not mutate canonical compose files. The blue service remains the canonical stack service; the green service is temporary.

Tier 8 Mapping

This runbook is the operator procedure behind the Tier 8 upgrade/version-skew plan:

• TOPO-130: service version skew uses the green sibling topology.
• TOPO-132: weighted canary uses weighted-canary-deployment.md and devops/observability/grafana/canary.json; this runbook remains the atomic blue/green path.
• TOPO-133: zero-downtime restart depends on Sprint 055 drain and Goodbye.
• TOPO-134: rollback-tag preservation remains the fallback when green is bad or image promotion fails.

Related files:

• Tier 8 upgrade/version-skew plan
• Weighted canary deployment runbook
• Sprint 055 graceful drain and forward-only migrations (completed; archived)
• Sprint 057 weighted canary (completed; archived)
• ADR-004 forward-only migrations
• Migration recovery runbook