Runbook — Authority / gateway auth troubleshooting

Applies to: deployed-stack auth failures (500/401 on authed calls, console-wide anonymous, CLI can’t get a token). Check both deployment wiring and the authentication response path; an unavailable metadata endpoint can expose a second response-handler defect. Canonical scope catalog: src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs. Verify against src/.

Credentials and current passwords are not in this doc (they’re operator-private). This documents the failure mechanisms and the reconcile fixes.

1. Authenticated calls fail while the service is healthy

Local Docker forwarding loop during an Authority outage

Use authority-upstream.stella-ops.local for Gateway’s internal Authority forwarding, metadata and credential-bearing requests. Compose declares this alias only on Authority’s container network; do not add it to the host’s hosts file. Keep the public issuer https://authority.stella-ops.local/ unchanged. The local wildcard certificate covers the upstream alias.

If the public hostname is also a host loopback alias, Docker DNS can resolve it to that loopback address while Authority is stopped. A Gateway HTTPS listener on all local addresses then forwards back into itself. Check from Gateway with getent hosts authority-upstream.stella-ops.local: it must resolve to Authority’s container IP when running and fail resolution when absent. Use direct readiness and token/discovery responses to verify recovery; a CPU limit increase does not repair the forwarding path.

After adding a confidential bootstrap client, provision its required secret in the existing ignored runtime environment using the installer’s secret-generation mechanism, render Authority’s recorded Compose chain, and recreate Authority plus the consuming service with --no-deps. Restarting keeps the old container environment. Inspect variable presence without printing its value.

Claims refresh also needs a Gateway binary containing ConfigureClaimsOverridesClient. Verify /app/buildinfo.json and its image digest: settings alone do not add the bearer handler to an older image. The implemented path authenticates as stellaops-gateway-claims, requests only authority:claims-overrides.read, and caches its token in process. Keep both the token endpoint and ROUTER_AUTHORITY_CLAIMS_OVERRIDES_URL on the internal HTTPS hostname with certificate validation enabled. Prove a successful refresh from Gateway’s own client and anonymous denial after deployment.

Cause: Authority__ResourceServer__MetadataAddress (or equivalent) points at plain http://. OpenIddict rejects all plain-HTTP discovery with HTTP 400 (IDX20807 — this server only accepts HTTPS requests) even with RequireHttpsMetadata=false. The client flag does not relax the server’s transport policy. The healthcheck is unauthenticated and may still report healthy.

Fix: verify a TLS endpoint from the caller’s network and keep the CA bundle mounted. In the 2026-09-21 Registry window, https://stella-ops.local/.well-known/openid-configuration and https://stella-ops.local/jwks both work, while the backend Authority hostname serves HTTP only. The resource server can set Authority:ResourceServer:MetadataAddress and optional JwksAddress to these explicit ingress endpoints. Keep Authority at the configured issuer, audience stellaops, and RequireHttpsMetadata=true. The override selects where authoritative public keys are fetched; it does not disable issuer, signature, audience or certificate checks. Changing JwksAddress invalidates cached keys.

The shared SDK maps metadata failures to a complete 503 Problem Details response with Retry-After (upstream timeouts use 504). OSM-7 found that the subsequent authorization challenge tried to set 401 after that response had started, aborting the body. Its pipeline regression reproduces the failure; the challenge now preserves the response already written by the failure mapper.

2. CLI can’t get a token

3. The whole console is anonymous (every route Actor=anonymous → tenant_missing)

Cause: the seeded stella-ops-ui client’s DB properties.audiences drifted and is missing stellaopswhile the gateway validates aud=stellaops. Every DPoP browser token then fails IDX10214 Audience validation failed at the gateway → principal anonymous → the tenant header is stripped and not re-injected → downstream 400 tenant_missing. It masquerades as a gateway tenant/ routing bug, and a Bearer curl with stellaops-cli (aud=stellaops) works the whole time, so token- level tests pass while the browser fails. Source is correct (standard.yaml includes stellaops); only long-lived DBs drift.

Reversible fix:

UPDATE authority.clients
SET properties = jsonb_set(properties,'{audiences}',
  '"stellaops api://export-center api://release-orchestrator api://issuer-directory notify"')
WHERE client_id='stella-ops-ui';

then docker restart stellaops-authority (clears the OpenIddict client cache) + a fresh login. Durable fix still OWED: a forward-only Authority reconcile that adds stellaops to persisted stella-ops-ui audiences for existing installs.

4. Authority doesn’t re-merge standard.yaml into the DB on restart

New scopes/audiences added to devops/etc/authority/plugins/standard.yaml are not reconciled into the bootstrapped authority.clients row on restart — the row goes stale (missing properties.audiences → empty-aud tokens → SecurityTokenInvalidAudienceException; missing newer scopes). Reconcile the live DB row (as in §3) or force a re-seed. A startup-reconcile would prevent the whole drift class.

5. Everything 401s right after a signing-key rotation

Symptom. Tokens that worked minutes ago return 401 everywhere; freshly minted tokens work. Nothing is unhealthy and nothing is in the logs, because this is not a fault — it is the rotation.

This is expected and correct. A token is only as valid as the key that signed it, so every token minted before the rotation instant dies at the rotation instant. Re-mint and continue. On the lab, the last rotation was 2026-08-21 (SPRINT_20260803_001 KEY-2R): dev-signing-key-1 → dev-signing-key-2, dev-ack-key-1 → dev-ack-key-2, keys moved to the git-ignored devops/etc/secrets/. Anything holding a token minted before ~06:10 UTC that day got a 401.

Tell it apart from a real fault in one command — read the kid out of the failing token and compare it with what Authority currently publishes:

# kid the token was signed with
cut -d. -f1 <<<"$TOKEN" | tr '_-' '/+' | base64 -d 2>/dev/null; echo
# kid(s) Authority serves right now
curl --fail --silent --show-error --cacert devops/compose/combined-ca-bundle.crt https://stella-ops.local/jwks | python -c "import json,sys; print([k['kid'] for k in json.load(sys.stdin)['keys']])"

Token kid absent from JWKS ⇒ stale token, re-mint. Token kid present and still 401 ⇒ this is not a rotation, keep reading at §1–§4.

If a rotation instead produced 401s on FRESHLY minted tokens, the kid was reused. That is the failure mode the kid-bump rule exists to prevent: a resource server caches JWKS by kid, so new key material under an unchanged kid means the consumer finds the entry it expects, never refetches, and rejects every token. Fix by rotating the kid too — see Authority architecture §6.

Topology notes

Confirm the fix (live forcing-function)

After a reconcile, prove it from the browser (a real DPoP login on the affected route), not just a Bearer curl — the curl path can stay green while the console is broken.