Weighted Canary Deployment Runbook

Audience: Stella Ops platform operators and SREs running progressive rollouts. Related: Blue/green compose deployment · devops/observability/grafana/canary.json

Overview

Use this runbook to introduce a new service version to a small percentage of live traffic, observe per-version behavior, and then promote or abort — without switching the whole service pool at once. For an atomic, one-version-at-a-time switch instead, see blue/green compose deployment; When To Use contrasts the two.

Router Gateway applies weights only after the normal routing filters have already selected routable candidates: health, draining state, version routing, region tier, and ping tie handling still apply first. Weighting then runs only inside the tied candidate bucket, and only when the tied instances do not all share the same effective weight (see How weighting is decided).

There are two ways to set weights, and both live on the gateway:

Runtime overrides via POST /api/v1/router/routing/weights. These are process-local to the gateway and are lost on a gateway restart.
Durable startup defaults via gateway config Router:Routing:WeightOverrides (a dictionary keyed by service/version or service/version/instanceId).

NOT IMPLEMENTED: there is no per-service Router__Microservice__Weight (or Router:Microservice:Weight) environment binding. The advertised weight a service announces in its HELLO (StellaMicroserviceOptions.Weight) is fixed at the default of 100 in the production registration path (StellaOps.Router.AspNet/StellaRouterIntegrationHelper.cs TryAddStellaRouter never reads a weight from configuration). To run a canary at a non-default weight, use the gateway runtime API or Router:Routing:WeightOverrides — not a container environment variable on the canary service.

When To Use

Decision point	Use weighted canary	Use blue/green compose
Rollout shape	Gradual N-percent exposure with live tuning.	Atomic switch with a rollback window.
Observation need	Compare request rate, error rate, latency, and instance counts by version.	Smoke test plus fixed bake window is enough.
Service state	Both versions can be active against the same backing stores.	The operator wants one active version at a time.
Rollback	Set the canary weight to 0 and drain it.	Restart blue and tear down green.
Blast radius	A small traffic slice is acceptable while the canary bakes.	Any canary exposure is unacceptable.

Do not use weighted canary for breaking schema migrations, authority, router-gateway, platform, postgres, valkey, or other singleton/infra services. For breaking migrations, follow ADR-004 and migration-recovery.md.

Pre-Flight

Confirm the graceful drain primitive is deployed on the baseline and canary image. The abort path depends on the Sprint 055 Draining/Goodbye behavior, not on killing containers.
```
docker exec stellaops-evidence-locker-web sh -lc \
  "grep -a -R -m1 'Stella microservice drain quiesce completed' /app 2>/dev/null"
```
Confirm Router Gateway metrics are present before changing traffic. The canary dashboard uses these Prometheus series:
```
curl -k https://stella-ops.local/metrics | grep stella_router_requests_total
curl -k https://stella-ops.local/metrics | grep stella_router_request_duration_seconds
curl -k https://stella-ops.local/metrics | grep stella_router_routing_weight
```
stella_router_routing_weight is a gauge that exposes the effective weight per service/version/instance; it backs the dashboard’s Routing weights (effective) panel and is the fastest way to confirm a weight override took effect.
Import devops/observability/grafana/canary.json into Grafana 10+ and point it at the Prometheus datasource that scrapes Router Gateway.
Provision an operator token with:
- router:routing:read for GET /api/v1/router/routing/weights
- router:routing:write for POST /api/v1/router/routing/weights
Record the logical Router service name, version, and instance IDs from the weight API. These can differ from compose service names.
```
curl -k -H "Authorization: Bearer $ROUTER_ROUTING_READ_TOKEN" \
  https://stella-ops.local/api/v1/router/routing/weights
```
Confirm the route under test admits both baseline and canary candidates. If callers pin a version to the baseline — via the X-Api-Version request header or an Accept: application/vnd.stellaops.v<N>+json media type — or Router Router:Routing:DefaultVersion forces the baseline version for this route, the canary will not receive traffic regardless of weight. (The version-pin header is X-Api-Version; there is no X-Stella-Service-Version header in the routing path. See StellaOps.Router.Gateway/Middleware/RoutingDecisionMiddleware.cs ExtractVersionFromRequest.)

SOP: Deploy Canary At Weight 10

The example below uses evidence-locker-web. Substitute another canary-eligible service only after reviewing the service owner checklist.

Start the canary instance from the N+1 image tag required by the release plan. The canary advertises the default weight of 100 in its HELLO — there is no per-service env var to advertise a lower weight, so the 90/10 split is applied at the gateway in step 3, not at the container. Use a local compose override (or release override) that only changes the image tag, and start it under a distinct container suffix so the baseline instance keeps running.
```
# docker-compose.canary.local.yml - local operator file, not committed
services:
  evidence-locker-web:
    image: stellaops/evidence-locker-web:<n+1-tag>
```
The compose container name for this service is stellaops-evidence-locker-web${INSTANCE_SUFFIX:-}, so INSTANCE_SUFFIX=-canary produces stellaops-evidence-locker-web-canary.
```
cd devops/compose
INSTANCE_SUFFIX=-canary docker compose -f docker-compose.stella-ops.yml \
  -f docker-compose.canary.local.yml up -d --no-deps evidence-locker-web
```

Re-read effective weights and capture the canary instanceId.

curl -k -H "Authorization: Bearer $ROUTER_ROUTING_READ_TOKEN" \
  https://stella-ops.local/api/v1/router/routing/weights \
  | tee ../../docs/qa/feature-checks/runs/latest-canary-weights.json

For an exact 90/10 one-baseline/one-canary split, set explicit runtime overrides for both instances. Each override resolves with the precedence instanceOverride > versionOverride > advertised instance weight, and the source field in the effectiveWeights response tells you which one applied (instanceOverride, versionOverride, or instance). If the baseline keeps its advertised weight of 100 and only the canary is set to 10, the split is 10 / (100 + 10) ≈ 9.1% to the canary. The accepted weight range is 0 to Router:Routing:MaxWeightOverride (default 1000); a POST outside that range returns 400.
```
curl -k -X POST https://stella-ops.local/api/v1/router/routing/weights \
  -H "Authorization: Bearer $ROUTER_ROUTING_WRITE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"service":"evidence-locker","version":"<baseline-version>","instanceId":"<baseline-instance-id>","weight":90}'

curl -k -X POST https://stella-ops.local/api/v1/router/routing/weights \
  -H "Authorization: Bearer $ROUTER_ROUTING_WRITE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"service":"evidence-locker","version":"<canary-version>","instanceId":"<canary-instance-id>","weight":10}'
```
Drive normal read traffic through the gateway. The gateway stamps routed responses with Server: stellaops/<service>/<version>, and the dashboard plots the same split from Router Gateway metrics.

Observe In Grafana

Open the imported Stella Ops Weighted Canary dashboard:

Per-version request rate should converge near the intended split after the first few minutes of traffic.
Per-version error rate should stay within the service owner’s error budget.
p50/p95/p99 latency for the canary must not regress beyond the release gate.
Per-instance request count should match the configured weights within normal random variation.
Routing weights (effective) plots stella_router_routing_weight so you can confirm the gateway is using the weights you set, independent of observed traffic.

Use a fixed bake window from the release plan. Fifteen minutes is the default for low-risk backend services; high-risk services need an owner-approved longer window.

Promote Or Abort

Promote when the canary remains inside the error and latency budget for the full bake window.

curl -k -X POST https://stella-ops.local/api/v1/router/routing/weights \
  -H "Authorization: Bearer $ROUTER_ROUTING_WRITE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"service":"evidence-locker","version":"<canary-version>","instanceId":"<canary-instance-id>","weight":100}'

Then either:

Drain the baseline pool and leave the canary version as the active pool.
Recreate the baseline pool from the promoted image tag, then drain the temporary canary instance after the canonical pool is healthy.

Abort when the canary violates the release gate, shows unexplained error-rate divergence, or has a version-specific data-shape issue.

curl -k -X POST https://stella-ops.local/api/v1/router/routing/weights \
  -H "Authorization: Bearer $ROUTER_ROUTING_WRITE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"service":"evidence-locker","version":"<canary-version>","instanceId":"<canary-instance-id>","weight":0}'

docker stop --time 30 stellaops-evidence-locker-web-canary

Rollback

Rollback does not require compose teardown to stop user traffic. Set the canary weight to 0, confirm the Grafana request-rate panel drops to zero for the canary instance, then drain the canary through the Sprint 055 shutdown path.

If the canary image ran a migration, do not attempt a reverse migration. Use the PostgreSQL snapshot restore path in migration-recovery.md.

Troubleshooting

How weighting is decided

Inside the tied candidate bucket, SelectFromTier only performs weighted selection when both of these hold:

At least one tied instance has an effective weight other than 100, and
the tied instances do not all share the same effective weight (there are at least two distinct values).

When those conditions are not met — for example every tied instance still advertises the default 100 — weighting is skipped and the configured tie-breaker decides: Router:Routing:TieBreaker is Random by default, with RoundRobin as the alternative. This is why a canary you started but never assigned a distinct weight to receives roughly an even share rather than a small slice.

The “tied” set itself requires an exact LastHeartbeatUtc match against the best candidate and an average ping within Router:Routing:PingToleranceMs (default 0.1 ms), so an instance with a newer heartbeat or higher ping never enters the weighted bucket at all.

Split looks uneven

Weighted selection only applies inside the tied candidate bucket. If the canary has a different region tier, degraded health, a newer heartbeat timestamp, or an average ping outside Router:Routing:PingToleranceMs, Router Gateway may select another candidate before weights are considered.

Actions:

Check GET /api/v1/router/routing/weights and confirm the canary appears in effectiveWeights with the expected effectiveWeight.
Confirm the canary is Healthy and not Draining.
Confirm the canary and baseline are in the same routing region tier.
If ping differs by more than PingToleranceMs, either fix the underlying network skew or temporarily widen Router:Routing:PingToleranceMs and restart Router Gateway.

The relevant code path is src/Router/__Libraries/StellaOps.Router.Gateway/Routing/DefaultRoutingPlugin.cs SelectFromTier.

Canary receives no traffic

Check for version pinning. A request that pins the baseline version — via the X-Api-Version header or an Accept: application/vnd.stellaops.v<N>+json media type — or a Router Router:Routing:DefaultVersion that excludes the canary version, will not enter the weighted bucket. There is no X-Stella-Service-Version header; the routing path reads version from X-Api-Version / the Accept media type only.

Weight API returns 401 or 403

Use a token with router:routing:read for GET and router:routing:write for POST. The endpoints are intentionally not anonymous.

Runtime weight disappears after gateway restart

Runtime overrides are process-local. Reapply them after a Router Gateway restart, or move the intended value into service config or Router:Routing:WeightOverrides before restarting.