Runbook — Authority / gateway auth troubleshooting

Applies to: deployed-stack auth failures (500/401 on authed calls, console-wide anonymous, CLI can’t get a token). These are recurring deployment/wiring drifts, not code defects — source is usually correct; long-lived DBs drift. Canonical scope catalog: src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs. Verify against src/.

Credentials and current passwords are not in this doc (they’re operator-private). This documents the failure mechanisms and the reconcile fixes.

1. Every authed call 500s with an empty body (service shows “healthy”)

Cause: Authority__ResourceServer__MetadataAddress (or equivalent) points at plain http://. OpenIddict rejects all plain-HTTP discovery with HTTP 400 (IDX20807 — this server only accepts HTTPS requests) even with RequireHttpsMetadata=false, so the JwtBearer handler can never fetch JWKS → the challenge fails after the response started → empty-body 500. The healthcheck is unauthenticated, so the container still reports healthy — the tell is “healthy service, every authed endpoint 500s”.

Fix: point it at https://router.stella-ops.local/.well-known/openid-configuration and set audience stellaops (policy-engine / scanner-worker already do this). In devops/compose/docker-compose.stella-services.yml, export-web/export-worker were fixed; other services may still carry the http:// form — sweep for it.

2. CLI can’t get a token

3. The whole console is anonymous (every route Actor=anonymous → tenant_missing)

Cause: the seeded stella-ops-ui client’s DB properties.audiences drifted and is missing stellaopswhile the gateway validates aud=stellaops. Every DPoP browser token then fails IDX10214 Audience validation failed at the gateway → principal anonymous → the tenant header is stripped and not re-injected → downstream 400 tenant_missing. It masquerades as a gateway tenant/ routing bug, and a Bearer curl with stellaops-cli (aud=stellaops) works the whole time, so token- level tests pass while the browser fails. Source is correct (standard.yaml includes stellaops); only long-lived DBs drift.

Reversible fix:

UPDATE authority.clients
SET properties = jsonb_set(properties,'{audiences}',
  '"stellaops api://export-center api://release-orchestrator api://issuer-directory notify"')
WHERE client_id='stella-ops-ui';

then docker restart stellaops-authority (clears the OpenIddict client cache) + a fresh login. Durable fix still OWED: a forward-only Authority reconcile that adds stellaops to persisted stella-ops-ui audiences for existing installs.

4. Authority doesn’t re-merge standard.yaml into the DB on restart

New scopes/audiences added to devops/etc/authority/plugins/standard.yaml are not reconciled into the bootstrapped authority.clients row on restart — the row goes stale (missing properties.audiences → empty-aud tokens → SecurityTokenInvalidAudienceException; missing newer scopes). Reconcile the live DB row (as in §3) or force a re-seed. A startup-reconcile would prevent the whole drift class.

5. Everything 401s right after a signing-key rotation

Symptom. Tokens that worked minutes ago return 401 everywhere; freshly minted tokens work. Nothing is unhealthy and nothing is in the logs, because this is not a fault — it is the rotation.

This is expected and correct. A token is only as valid as the key that signed it, so every token minted before the rotation instant dies at the rotation instant. Re-mint and continue. On the lab, the last rotation was 2026-08-21 (SPRINT_20260803_001 KEY-2R): dev-signing-key-1 → dev-signing-key-2, dev-ack-key-1 → dev-ack-key-2, keys moved to the git-ignored devops/etc/secrets/. Anything holding a token minted before ~06:10 UTC that day got a 401.

Tell it apart from a real fault in one command — read the kid out of the failing token and compare it with what Authority currently publishes:

# kid the token was signed with
cut -d. -f1 <<<"$TOKEN" | tr '_-' '/+' | base64 -d 2>/dev/null; echo
# kid(s) Authority serves right now
curl -sk https://stella-ops.local/jwks | python -c "import json,sys; print([k['kid'] for k in json.load(sys.stdin)['keys']])"

Token kid absent from JWKS ⇒ stale token, re-mint. Token kid present and still 401 ⇒ this is not a rotation, keep reading at §1–§4.

If a rotation instead produced 401s on FRESHLY minted tokens, the kid was reused. That is the failure mode the kid-bump rule exists to prevent: a resource server caches JWKS by kid, so new key material under an unchanged kid means the consumer finds the entry it expects, never refetches, and rejects every token. Fix by rotating the kid too — see Authority architecture §6.

Topology notes

Confirm the fix (live forcing-function)

After a reconcile, prove it from the browser (a real DPoP login on the affected route), not just a Bearer curl — the curl path can stay green while the console is broken.