Runbook — Authority / gateway auth troubleshooting

Applies to: deployed-stack auth failures (500/401 on authed calls, console-wide anonymous, CLI can’t get a token). These are recurring deployment/wiring drifts, not code defects — source is usually correct; long-lived DBs drift. Canonical scope catalog: src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs. Verify against src/.

Credentials and current passwords are not in this doc (they’re operator-private). This documents the failure mechanisms and the reconcile fixes.

1. Every authed call 500s with an empty body (service shows “healthy”)

Cause: Authority__ResourceServer__MetadataAddress (or equivalent) points at plain http://. OpenIddict rejects all plain-HTTP discovery with HTTP 400 (IDX20807 — this server only accepts HTTPS requests) even with RequireHttpsMetadata=false, so the JwtBearer handler can never fetch JWKS → the challenge fails after the response started → empty-body 500. The healthcheck is unauthenticated, so the container still reports healthy — the tell is “healthy service, every authed endpoint 500s”.

Fix: point it at https://router.stella-ops.local/.well-known/openid-configuration and set audience stellaops (policy-engine / scanner-worker already do this). In devops/compose/docker-compose.stella-services.yml, export-web/export-worker were fixed; other services may still carry the http:// form — sweep for it.

2. CLI can’t get a token

The only reliable token against the lab = password grant with authority_provider=standard(the param is authority_provider, NOT identity_provider; required because the gateway has multiple password providers) via client_id=stellaops-cli at the gateway-root /connect/token(not /authority/connect/token).
The human stellaops-cli client has no secret (client_credentials → 400); some automation clients may be “Unknown or disabled” in a given deployed Authority. Grants are per-DB — stellaops-cli may only carry e.g. export.*, not notify.*.
response.scope from the token endpoint is empty — decode the JWT scope claim to confirm what you actually got. Working recipe: devops/compose/scripts/compliance-setup.sh.

3. The whole console is anonymous (every route Actor=anonymous → `tenant_missing`)

Cause: the seeded stella-ops-ui client’s DB properties.audiences drifted and is missing stellaopswhile the gateway validates aud=stellaops. Every DPoP browser token then fails IDX10214 Audience validation failed at the gateway → principal anonymous → the tenant header is stripped and not re-injected → downstream 400 tenant_missing. It masquerades as a gateway tenant/ routing bug, and a Bearer curl with stellaops-cli (aud=stellaops) works the whole time, so token- level tests pass while the browser fails. Source is correct (standard.yaml includes stellaops); only long-lived DBs drift.

Reversible fix:

UPDATE authority.clients
SET properties = jsonb_set(properties,'{audiences}',
  '"stellaops api://export-center api://release-orchestrator api://issuer-directory notify"')
WHERE client_id='stella-ops-ui';

then docker restart stellaops-authority (clears the OpenIddict client cache) + a fresh login. Durable fix still OWED: a forward-only Authority reconcile that adds stellaops to persisted stella-ops-ui audiences for existing installs.

4. Authority doesn’t re-merge `standard.yaml` into the DB on restart

New scopes/audiences added to devops/etc/authority/plugins/standard.yaml are not reconciled into the bootstrapped authority.clients row on restart — the row goes stale (missing properties.audiences → empty-aud tokens → SecurityTokenInvalidAudienceException; missing newer scopes). Reconcile the live DB row (as in §3) or force a re-seed. A startup-reconcile would prevent the whole drift class.

Topology notes

stellaops-authority runs its own PostgreSQL container, separate from stellaops-postgres. Auth data lives there.
Under disk/memory pressure the authority container overlay can hit a docker exec … no such file or directory overlayfs glitch → docker exec/healthcheck fail (“unhealthy”) while the process serves fine (discovery returns 200). A recreate clears it.
Gateway identity envelope: the gateway signs an envelope (HMAC-SHA256, key in .env STELLAOPS_IDENTITY_ENVELOPE_SIGNING_KEY) carrying subject/tenant/scopes/roles; services verify it. If a claim isn’t in the envelope’s allow-list it won’t reach the service (e.g. stellaops:admin must be explicitly forwarded for global-admin console checks).

Confirm the fix (live forcing-function)

After a reconcile, prove it from the browser (a real DPoP login on the affected route), not just a Bearer curl — the curl path can stay green while the console is broken.