Runbook — Authority break-glass recovery

Use this when: an operator is locked out of Authority, or Authority itself is broken, and no normal admin surface can fix it. This is the estate’s most privileged path. Every use is recorded, expires, and shows up as a Critical doctor finding until someone acknowledges it.

Sprint: SPRINT_20260722_016 AUTH-11. Design: docs/modules/authority/break-glass-access-design.md(owner sign-off 2026-08-14).

What this is, and what it is not

Break-glass here is not a network endpoint, a bootstrap flag, or a standing emergency account. It is the stella admin users password-set --confirm-authority-access tool, run as a one-shot container that opens Authority’s database directly and resets one user’s credential.

That shape is deliberate, and the argument is worth knowing before you reach for something else:

• Its authentication factor is possession of the database credential plus shell on the host. There is no listener to misconfigure. The estate has already shipped the opposite mistake — a “localhost-only” surface reachable one ASPNETCORE_URLS or proxy rule away.
• It works when Authority does not. A recovery path hosted inside the thing that is broken is not a recovery path. During the 2026-07-29 → 08-10 freeze Authority crash-looped at exit 139 on a migration checksum mismatch; no in-process endpoint would have answered.

It is not a way to create an administrator. It resets an EXISTING user’s password, the credential expires (default 15 minutes), and the use is on the record.

Before you start

You need:

1. Shell on the host running the compose stack.
2. STELLAOPS_POSTGRES_AUTHORITY_CONNECTION set in the compose .env (the tool refuses to run without an Authority storage environment marker — that is its environment gate).
3. A reason. --reason is required. It is written into the audit ledger and it is the only field that answers the question a post-incident review actually asks.
4. The username and tenant. Break-glass is restricted to structural tenants — installation and default. A per-tenant password reset is a support question answered through the normal admin surface, not through the auth root; the tool refuses other tenants and names the permitted ones.

Procedure

cd devops/compose

# Linux/macOS
./scripts/authority-password-set.sh admin \
  --tenant default \
  --reason "INC-1234: operator locked out after failed logins" \
  --password-stdin <<< 'NewStrongPassword@2026!'

# Windows
./scripts/authority-password-set.ps1 -Username admin `
  -Tenant default `
  -Reason "INC-1234: operator locked out after failed logins" `
  -PasswordStdin

Password policy is enforced by the tool before it touches the database: at least 12 characters with upper, lower, digit and symbol.

On success it prints the expiry and tells you the use was recorded:

Set password for Authority user 'admin' in tenant 'default'. Lockout counters were cleared.
This credential EXPIRES at 2026-08-18 09:15:00Z (900s). Change the password through Authority before then.
The use was recorded in authority.break_glass_events and doctor will report CRITICAL until it is reconciled.

Then, immediately

1. Log in and change the password through Authority. The break-glass credential is entry, not an account. When it expires the credential stops working — the grant path enforces password_expires_at, so walking away leaves an expired credential rather than a permanent administrator.
2. Reconcile the event (below). Until you do, doctor.authority.break-glass.unreconciled reports Critical.

If you need longer than 15 minutes

--expires-in-seconds 3600

Bounded to 60 … 86400. The value you choose is stamped into the audit row, so a stretched window is visible after the fact rather than invisible. The air-gapped case is why this is configurable at all: an operator with no console may genuinely need longer.

Reconciling a use

This is part of the procedure, not an optional tidy-up. The doctor check clears on explicit acknowledgement and never on a timer — a Critical that ages out would train operators to wait rather than reconcile.

# What is outstanding?
stella admin break-glass list --unreconciled-only

# Acknowledge one
stella admin break-glass reconcile <event-id> \
  --by "ops-lead@example" \
  --note "reviewed at incident standup; password rotated"

--by is required: an acknowledgement that names nobody clears a Critical without attaching anyone to the decision.

Reconciliation is recorded once. A second attempt is refused rather than overwriting who accepted it — enforced by a database trigger, not just by the CLI.

What the ledger records, and what it cannot

authority.break_glass_events carries, per use: when, which tenant and subject, the actor (OS user + host — break-glass runs when Authority cannot authenticate anyone, so the operating system’s identity is the only one available), the reason, the tool version, the resolved expiry and the configured window, and the reconciliation.

Three properties worth knowing:

• The audit insert happens FIRST, in the same transaction as the credential change. If the audit write fails, the password change does not happen. An unauditable break-glass is an unusable one — the other ordering makes the audit advisory, and the first person who needs it to be advisory will make it so.
• Rows are append-only. DELETE always raises; UPDATE may move only the reconciliation columns, once. Enforced by trigger.
• The ledger starts at migration 025. An empty table means nothing has been recorded since then — it is not a claim that break-glass has never been used. There was no record before it, and none can be recovered.

Doctor

doctor.authority.break-glass.unreconciled (category data-integrity):

Reaching it: GET /doctor/authority/checks on authority, scope-gated on ops.health.

The drill

The scenario is exercised end-to-end by StellaOps.Authority.Plugin.Standard.Tests.BreakGlass.BreakGlassRecoveryDrillTests against a real PostgreSQL with Authority’s real migration chain: converge a fresh database, log in, manufacture a real lockout through repeated failed logins (not by writing locked_until), confirm login fails, recover through the shipped break-glass store, confirm login succeeds and exactly one audit row exists and is undeletable, then advance the clock past the expiry and confirm the recovered credential stops working.

That last step is the one most likely to be dropped and the only one that tests the constraint the pre-hardening mechanism most clearly violated.

Related

• Design and the security argument: break-glass-access-design.md
• Bootstrap and first-admin flows: ../modules/authority/operations/bootstrap-modes.md
• Not ../modules/authority/operations/break-glass-account.md — that document describes a stella auth break-glass command and a POST /auth/break-glass endpoint that do not exist; the BreakGlassSessionManager design it describes was removed in SPRINT_20260731_003. It is superseded by this runbook.