Break-Glass Account Runbook

Audience: Stella Ops platform operators and security/incident responders. Purpose: Document the emergency-access design for break-glass accounts — how an operator regains administrative access to the Authority service when standard authentication (OIDC, the primary policy store, or the Authority service itself) is unavailable.

This runbook describes the Stella Ops break-glass subsystem: configuration, login and session lifecycle, the audit trail, and the fallback policy store. Read it before an incident so the procedure is familiar when you need it.

Sprint: SPRINT_20260112_018_AUTH_local_rbac_fallback

STATUS — DRAFT / NOT WIRED (verified against src/ 2026-05-30). The break-glass subsystem exists as library code in src/Authority/StellaOps.Authority/StellaOps.Authority/LocalPolicy/ (ILocalPolicyStore/FileBasedPolicyStore, IBreakGlassSessionManager/BreakGlassSessionManager, FallbackPolicyStore, LocalPolicyModels), covered by unit/integration tests. However, as of this reconciliation it is not registered in the Authority service’s DI container, has no HTTP login endpoint, no CLI command, no Console UI surface, and emits no Prometheus metrics. The classes are reachable only from their own tests. Treat the procedures below as the intended design / forward roadmap, not a deployed, operator-usable feature. Sections that describe surfaces which do not yet exist in code are marked (NOT IMPLEMENTED) inline. Do not rely on this runbook for live incident response until the wiring tasks land.

Overview

Break-glass accounts are intended to provide emergency administrative access when:

• PostgreSQL database is unavailable
• OIDC/OAuth2 identity provider is unreachable
• Authority service is degraded
• Network isolation prevents standard authentication

The design audits all break-glass activity (via an IBreakGlassAuditLogger) and time-limits every session.

When to Use Break-Glass Access

CRITICAL: Break-glass access should only be used when standard authentication is genuinely unavailable. All usage is logged and auditable.

Prerequisites

Configuration Requirements

Break-glass is configured in the local policy file. The file path is set by LocalPolicyStoreOptions.PolicyFilePath (config section Authority:LocalPolicy) and defaults to /etc/stellaops/authority/local-policy.yaml. Both YAML and JSON are supported — FileBasedPolicyStore selects the parser by file extension (.yaml/.yml or .json). Property names are camelCase in both formats.

# /etc/stellaops/authority/local-policy.yaml
schemaVersion: "1.1.0"        # required; supported: 1.0.0, 1.0.1, 1.1.0
lastUpdated: "2026-01-16T00:00:00Z"  # required
breakGlass:
  enabled: true
  sessionTimeoutMinutes: 15     # default 15
  maxExtensions: 2              # default 2
  requireReasonCode: true       # default true
  allowedReasonCodes:           # default: EMERGENCY, INCIDENT, DISASTER_RECOVERY, SECURITY_EVENT, MAINTENANCE
    - EMERGENCY
    - INCIDENT
    - DISASTER_RECOVERY
    - SECURITY_EVENT
    - MAINTENANCE
  accounts:
    - id: "break-glass-admin"
      displayName: "Emergency Admin"
      credentialHash: "$2b$12$..."   # bcrypt hash; field is credentialHash, NOT passwordHash
      hashAlgorithm: "bcrypt"        # default "bcrypt" — see note below
      roles: ["admin"]
      enabled: true                  # default true
      # expiresAt: "2026-12-31T00:00:00Z"  # optional account expiry

Reason codes: the default allowedReasonCodes are the literal values EMERGENCY, INCIDENT, DISASTER_RECOVERY, SECURITY_EVENT, MAINTENANCE (see BreakGlassConfig.AllowedReasonCodes). They are compared case-insensitively. You may override this list, but the codes below in this runbook reflect those defaults. Matching is enforced only when requireReasonCode is true.

Password Hash Generation

Hashing — bcrypt only (verified in FileBasedPolicyStore.VerifyCredentialHash). The credential verifier currently supports only bcrypt(BCrypt.Net.BCrypt.Verify). The argon2id branch is present but commented out in source, so an account configured with hashAlgorithm: "argon2id" will fail verification (the switch returns false for any non-bcrypt algorithm). Use bcrypt hashes until the argon2id path is implemented.

Generate a bcrypt credential hash with any standard bcrypt tool, for example:

# Using htpasswd (bcrypt, cost 12)
htpasswd -nbB break-glass-admin "your-secure-password" | cut -d: -f2

# Using python bcrypt
python3 -c 'import bcrypt; print(bcrypt.hashpw(b"your-secure-password", bcrypt.gensalt(12)).decode())'

(NOT IMPLEMENTED) There is no stella auth hash-password CLI command — the CLI exposes no auth command group. Generate hashes with an external bcrypt utility as shown above.

Break-Glass Login Procedure

Step 1: Verify Standard Auth is Unavailable

Before using break-glass, confirm standard authentication is genuinely unavailable:

# Check Authority health
curl -s https://authority.example.com/health | jq .

# Check OIDC endpoint
curl -s https://idp.example.com/.well-known/openid-configuration

Step 2: Access Break-Glass Login

(NOT IMPLEMENTED) There is no /break-glass/login HTTP endpoint and no stella auth break-glass login CLI command in the current build. Session creation is exposed only as the in-process API IBreakGlassSessionManager.CreateSessionAsync(...). The intended invocation surfaces (HTTP endpoint / CLI / Console) are part of the forward roadmap for this sprint and are documented here for design reference.

The design contract for creating a session is IBreakGlassSessionManager.CreateSessionAsync, which takes a BreakGlassSessionRequest.

Step 3: Provide Credential and Reason

Note: the request is authenticated by credential alone — there is no separate “account ID” input. CreateSessionAsync resolves the matching account by verifying the supplied credential against every enabled, unexpired account in breakGlass.accounts. The fields on BreakGlassSessionRequest are:

Default Approved Reason Codes (override via allowedReasonCodes):

Step 4: Session Created

On successful authentication (BreakGlassSessionResult.Success == true):

• A BreakGlassSession is created in-memory with ExpiresAt = now + sessionTimeoutMinutes (default 15 minutes) and a cryptographically random base64url SessionId (32 bytes).
• The session carries the resolved account’s roles.
• An audit event of type SessionCreated is emitted via IBreakGlassAuditLogger.
• The event is logged at Warning level (the only shipped logger is ConsoleBreakGlassAuditLogger, described as “for development/fallback”).

Session Management

Session Timeout

Break-glass sessions have strict time limits:

On each successful extension, ExpiresAt is reset to now + sessionTimeoutMinutes (it does not add a separate fixed “extension period”). Expired sessions are reaped by a background cleanup timer that runs once per minute.

Extending a Session

Extension is the in-process API IBreakGlassSessionManager.ExtendSessionAsync(sessionId, credential).

(NOT IMPLEMENTED) No stella auth break-glass extend CLI command or Console “Extend Session” banner exists in the current build.

Extension requires:

1. Re-supplying the break-glass credential (re-validated against the policy store)
2. The session not having reached maxExtensions (otherwise MAX_EXTENSIONS_REACHED)

Note: the ExtendSessionAsync contract does not take a free-text “extension reason” parameter — it takes only the session ID and the credential.

Session Termination

Sessions end when:

• The session is explicitly terminated (TerminateSessionAsync)
• The session timeout expires (reaped by the cleanup timer; SessionExpired audit event)
• Max extensions reached (further extension attempts fail)

Termination is the in-process API IBreakGlassSessionManager.TerminateSessionAsync(sessionId, reason).

(NOT IMPLEMENTED) No stella auth break-glass logout / terminate CLI commands exist in the current build.

Audit Trail

Audit Events

All break-glass activity is logged through IBreakGlassAuditLogger. The event types are the BreakGlassAuditEventType enum values (not dotted strings):

Audit Event Structure

The BreakGlassAuditEvent record fields are:

{
  "eventId": "f3a2...",            // GUID "N" format
  "eventType": "SessionCreated",   // BreakGlassAuditEventType
  "timestamp": "2026-01-16T10:30:00Z",
  "sessionId": "bg-...",           // base64url, 32 random bytes (nullable)
  "accountId": "break-glass-admin",
  "reasonCode": "MAINTENANCE",
  "reasonText": "PostgreSQL major version upgrade",  // field is reasonText, not reasonDetails
  "clientIp": "10.0.1.50",         // field is clientIp, not sourceIp
  "userAgent": "stella-cli/...",
  "details": { "message": "..." }  // optional free-text map
}

Querying Audit Logs

(NOT IMPLEMENTED) The only shipped audit sink is ConsoleBreakGlassAuditLogger, which writes a [BREAK-GLASS-AUDIT] Warning-level log line; it is described in source as “for development/fallback”. There is no DB-backed break-glass audit store and no stella audit query / stella audit export integration that surfaces these events. To retrieve break-glass activity today, search the Authority service logs for [BREAK-GLASS-AUDIT], e.g.:

# Authority container/service logs
grep "\[BREAK-GLASS-AUDIT\]" /var/log/stellaops/authority.log

Fallback Policy Store

The composite FallbackPolicyStore (config section Authority:PolicyFallback) is the design for switching between the PostgreSQL-backed primary store and the local file store.

STATUS: Like the rest of this subsystem, FallbackPolicyStore and its IPrimaryPolicyStoreHealthCheck are present as classes with tests but are not wired into the live Authority service (no DI registration). The behaviour below is the implemented class behaviour, not a confirmed live failover path.

Automatic Failover

When the primary store health check fails:

1. A background timer probes the primary every HealthCheckIntervalMs (default: 5000 ms)
2. After FailureThreshold (default: 3) consecutive failures
3. The store switches to the local policy store (SwitchToFallback)
4. CurrentMode changes to Fallback
5. A ModeChanged .NET event is raised and a Warning is logged. (Note: this is an in-process event, not an authority.mode.changed audit record — no such audit event type exists in code.)

Policy Store Modes

The PolicyStoreMode enum has three values:

Note: the enum defines Degraded, but the current FallbackPolicyStore logic only transitions between Primary and Fallback; it does not set Degraded itself. The “Available Features” per mode (full RBAC vs. break-glass only) is a design intent, not enforced branching in this class.

Recovery

When the primary store becomes healthy again:

1. Health checks pass (_consecutiveFailures resets to 0)
2. After at least MinFallbackDurationMs (default: 30000 ms / 30 s) in fallback mode
3. The store switches back to Primary (SwitchToPrimary), raising ModeChanged
4. Any in-memory break-glass sessions continue until their own expiry

Security Considerations

Password Policy

Break-glass account passwords should:

• Be at least 20 characters
• Include upper, lower, numbers, symbols
• Be hashed with bcrypt (the only verifier currently implemented) before placing the credentialHash in the policy file
• Be stored securely (Vault, split custody; on-prem first — no cloud-managed KMS defaults)
• Be rotated on a schedule (quarterly recommended)

Access Control

• Limit break-glass accounts to essential personnel
• Use separate accounts per operator when possible
• Review access list quarterly
• Disable unused accounts immediately

Monitoring

(NOT IMPLEMENTED) The break-glass code path emits no Prometheus metrics — there is no stellaops_breakglass_sessions_created_total counter (or any break-glass metric) in source. Until metrics are added, alert on the structured log line emitted by ConsoleBreakGlassAuditLogger ([BREAK-GLASS-AUDIT] EventType=SessionCreated ...) via your log-based alerting pipeline.

The example below is the intended metric-based alert once a counter exists, kept for roadmap reference:

# Alert rule example (metric NOT yet emitted — see note above)
- alert: BreakGlassSessionCreated
  expr: stellaops_breakglass_sessions_created_total > 0
  for: 0m
  labels:
    severity: warning
  annotations:
    summary: Break-glass session created
    description: A break-glass session was created. Verify this is expected.

Troubleshooting

Login / Session Result Error Codes

These are the ErrorCode values returned on BreakGlassSessionResult (from BreakGlassSessionManager):

Note: an account that is disabled or expired is simply skipped during credential matching, so it surfaces as AUTHENTICATION_FAILED rather than a distinct “account disabled” code.

Session Issues

Policy Store Issues

(NOT IMPLEMENTED) There are no stella auth policy validate / stella auth policy reload CLI commands (the CLI has no auth command group). Policy reload happens automatically via the FileBasedPolicyStore hot-reload file watcher (when EnableHotReload is true, default), or in-process via ILocalPolicyStore.ReloadAsync. To force a reload operationally, modify/re-save the policy file so the watcher picks it up, or restart the Authority service. Validation feedback (schema version, signature, role/subject counts) is surfaced through the PolicyReloaded event and the Authority service logs.

Compliance Notes

Break-glass usage must be:

• Documented in incident reports
• Reviewed during security audits
• Reported in compliance dashboards
• Justified for each session

Retain audit logs for:

• SOC 2: 1 year minimum
• HIPAA: 6 years
• PCI-DSS: 1 year
• Internal policy: As defined

Related Documentation

• Break-Glass Account Operations — sibling module-local operations note for the same subsystem
• Authority Architecture
• Air-Gap Operations Runbook
• Audit Emission Guide
• Local RBAC policy & break-glass source of truth: src/Authority/StellaOps.Authority/StellaOps.Authority/LocalPolicy/