Runbook: Policy Engine - Policy Storage Backend Down

For platform on-call. Use this when Stella Ops policy operations fail with storage errors, the Policy Engine crash-loops on startup migrations, or Doctor reports storage_status: FAILED. The Policy Engine persists exclusively to a PostgreSQL policy schema (with a Valkey cache) — there is no MongoDB store and no second backend to fail over to — so “storage unavailable” almost always means PostgreSQL on the Policy DSN is unreachable, misconfigured, or out of capacity. Triage it like any other Stella service whose Postgres is down.

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-003 - Policy Engine Runbooks

Reconciliation note (verify against src/Policy): The original draft of this runbook described a generic “policy storage backend” with a primary/backup failover model, an OPA bundle loader, and a large family of stella policy storage … / stella db … commands. That does not match the implementation. The Policy Engine (src/Policy/StellaOps.Policy.Engine) persists exclusively to a PostgreSQL schema named policy(plus a policy_app role schema), owned by StellaOps.Policy.Persistence and applied via forward-only startup migrations (AddStartupMigrations("policy", "Policy.Persistence", …) wired in PolicyPersistenceExtensions; schema name forced to policy in PolicyDataSource.DefaultSchemaName). There is no MongoDB store, no second “backup storage” to fail over to, and OPA is not the storage backend — OPA is only an optional Rego interop/attestation-gate surface (see policy-opa-crash.md). EPSS/risk score rows are durable in the Policy DSN under policy.*. The evaluation/overlay/exception caches go to a pluggable distributed cache (IDistributedCache<T>; MessagingPolicyEvaluationCache supports InMemory, Valkey, or PostgreSQL transports) — Valkey is the default service cache per the module README (src/Policy/README.md: “Valkey for cache”). Only when the messaging Postgres transport is selected do those caches land in PostgreSQL, where the transport names them messaging.cache_<name> (default schema messaging; see PostgresCacheStore.cs). So “policy storage unavailable” almost always means PostgreSQL on the Policy DSN is unreachable, misconfigured, or out of capacity — handle it like any other Stella service whose Postgres is down.

Commands, alerts, and check IDs that do not exist in src/ are flagged inline as NOT IMPLEMENTED. Verified sources are listed under Related Resources.

Metadata

FieldValue
ComponentPolicy Engine (PostgreSQL policy schema + Valkey cache)
SeverityCritical
On-call scopePlatform team
Last updated2026-01-17 (reconciled against src/Policy)
Doctor checkcheck.policy.engine (source: src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Policy/Checks/PolicyEngineHealthCheck.cs)

Flag (corrected): The original check.policy.storage-health Doctor check does not exist. The Policy Engine plugin registers exactly one check, check.policy.engine. Its evidence includes a storage_status sub-result (OK / FAILED) and a storage_error field, so it is the verified way to observe Policy storage health from Doctor. (Note: that check probes a policy-engine URL with OPA-style endpoints — /v1/data for the storage probe — so it reflects the engine answering, not a direct Postgres ping. Use stella db status for the database view.)

Symptoms

Flag (NOT IMPLEMENTED): The original symptom “Alert PolicyStorageUnavailable firing” references an alert that is not defined anywhere in src/ or devops/. Do not page on it. If you want alerting, wire it against PostgreSQL availability for the Policy DSN (your Postgres monitoring) and/or the check.policy.engine Doctor result — not a Stella-emitted PolicyStorage* series.

Flag (corrected): “OPA unable to load bundles from storage” does not apply to the core engine. Policy packs are reconstructed by recompiling the persisted DSL source from PostgreSQL (policy.packs / policy.pack_versions) on reload — there is no OPA bundle store in the storage path. OPA bundle loading is a separate interop concern covered by policy-opa-crash.md.

Impact

Impact TypeDescription
User-facingPolicy authoring/lifecycle writes (pack create/activate/publish, exceptions, receipts) fail while Postgres is unreachable. Evaluation of already-activated packs can continue from recompiled-source/IR and cached EPSS/risk rows until those need refresh.
Data integrityNo corruption from an outage alone: writes fail loudly rather than persisting partially. Risk is operating on stale cached EPSS/risk values (rows carry expires_at, default NOW() + 7 days) if the outage is prolonged.
SLA impactPolicy management is blocked; release gates that require a fresh policy evaluation or exception write will block (or fail-open per their own gate config) until storage recovers.

Diagnosis

Quick checks

  1. Check Doctor diagnostics (verified check ID):

    stella doctor --check check.policy.engine

    Look for storage_status, storage_error, and engine_url in the evidence. A storage_status: FAILED points here; a connection_error_type: refused/timeout on the whole check means the engine itself is unreachable.

  2. Check database connectivity from the CLI:

    stella db status            # add --verbose for the connection-pool view, --format json for scripting

    stella db status queries GET /api/v1/health/database and reports connection state, database type/version, schema/migration status, and (with --verbose) pool counters. Source: src/Cli/StellaOps.Cli/Commands/DbCommandGroup.cs.

    Caveat (verify against DbCommandGroup.cs): if the API call itself fails (HttpRequestException), the CLI falls back to a synthetic “healthy” status (GenerateSyntheticStatus(), lines ~138/155) rather than surfacing the failure. A clean “Connected / up-to-date” from stella db status when the service is clearly down may be this fallback, not a real probe. Cross-check with the service health endpoint and container logs before trusting a green result.

  3. Inspect the Policy Engine service logs with your container/host tooling (there is no Stella log subcommand — see flag below):

    docker logs --since 30m stellaops-policy-engine
# or: journalctl -u <policy-engine-unit> --since '30 min ago'

    Look for migration errors, Npgsql connection failures, or pool-exhaustion timeouts on the Postgres:Policy DSN.

Flag (NOT IMPLEMENTED): stella policy storage status, stella db status --component policy, and stella db status --database policy do not exist. stella db status takes only --format, --server, and --verbose (no --component/--database); it reports the platform database health, not a per-module store. stella policy logs --filter storage … is also not a command — use container/host logs.

Deep diagnosis

  1. Confirm the Policy DSN and pool settings. The engine binds Postgres:Policy (PostgresOptions): ConnectionString, MaxPoolSize, MinPoolSize, CommandTimeoutSeconds, ConnectionIdleLifetimeSeconds, Pooling, AutoMigrate. The schema is forced to policy regardless of the shared options slot (PolicyDataSource.CreateOptions). Verify the connection string resolves and the role can reach the policy schema.

  2. Check PostgreSQL itself with standard Postgres tooling against the Policy DSN host (server up? accepting connections? disk full? too many connections? role/permissions intact?). This is ordinary Postgres triage — there is no Stella-specific ping/pool command (see flag).

  3. Check migration state. stella db status reports migrationStatus and pendingMigrations. A non-up-to-date status with the service crash-looping on boot indicates the startup migrator could not apply Migrations/001_initial_schema.sql … 015_*.sql. 001_initial_schema.sql is idempotent (CREATE … IF NOT EXISTS) and converges on reused volumes, so a genuine failure here is usually connectivity/permissions, not a half-applied schema.

Flag (NOT IMPLEMENTED): stella db ping --database policy, stella db pool-status --database policy, and stella policy storage disk-usage do not exist. Use stella db status --verbose for pool counters and your Postgres/host tooling for ping and disk usage.

Resolution

Immediate mitigation

  1. Restore PostgreSQL on the Policy DSN. The Policy Engine has a single durable store; there is no second backend to fail over to. Bring Postgres back (restart the database, free connections, free disk) — that is the mitigation.

    Flag (NOT IMPLEMENTED): stella policy storage failover --to backup and a “switch to backup storage” model do not exist. There is no configured backup store; recovery is restoring the primary PostgreSQL instance (HA/replica promotion, if you run one, is handled at the PostgreSQL layer, not via Stella).

  2. Let cached evaluation carry read traffic while Postgres recovers. Activated packs evaluate from recompiled source/IR and cached EPSS/risk rows; you do not need a read-only toggle to keep evaluating already-active policy.

    Flag (NOT IMPLEMENTED): stella policy config set storage.read_only true and stella policy reload are not commands. There is no global storage.read_only config key. (Pack reload happens automatically on engine restart by recompiling persisted DSL source.)

  3. Restart the Policy Engine to reconnect once Postgres is healthy, using your normal deployment tooling:

    docker restart stellaops-policy-engine

    On boot the engine re-runs startup migrations (idempotent) and re-establishes the connection pool.

    Flag (NOT IMPLEMENTED): stella service restart policy-engine is not a CLI command. Restart the stellaops-policy-engine container/service with your orchestration tooling (compose/systemd/etc.).

Root cause fix

If it is a database connection / pool issue:

  1. Inspect status and pool counters:

    stella db status --verbose

  2. Tune pooling via configuration, not a CLI mutation. Adjust Postgres:Policy (MaxPoolSize, MinPoolSize, CommandTimeoutSeconds) in the Policy Engine configuration and restart the service to apply.

    Flag (NOT IMPLEMENTED): stella db pool-restart --database policy and stella db config set policy.max_connections 50 are not commands. Pool sizing is configuration (Postgres:Policy.MaxPoolSize) applied on restart; Postgres-side max_connections is tuned on PostgreSQL itself.

If disk space is exhausted (on the PostgreSQL host):

  1. Free space on the database host with standard PostgreSQL/OS tooling (VACUUM/pg_repack as appropriate, log/WAL cleanup, volume growth). Policy tables of interest are under the policy schema.

    Flag (NOT IMPLEMENTED): stella policy storage disk-usage [--verbose] and stella policy versions cleanup --older-than 30d do not exist. Pack version retention is managed through the verified lifecycle commands (stella policy rollback, stella policy history) and at the database/data layer — there is no versions cleanup command.

If the schema/migrations are out of step:

  1. Confirm stella db status shows the expected migrationStatus. If the service could not apply migrations because of permissions, grant the role access to the policy schema and restart so AddStartupMigrations can converge. Platform policy is forward-only (see ADR-004); recover a bad downgrade by PostgreSQL snapshot restore (see the migration-recovery runbook), not by hand-editing tables.

    Flag (NOT IMPLEMENTED): stella policy storage verify and stella policy storage restore --from-backup latest do not exist. There is no Stella-side storage verify/restore for the Policy schema; integrity recovery is PostgreSQL backup/restore at the database layer.

Verification

# Database reports connected and migrations up-to-date
stella db status --verbose

# Doctor's Policy check passes (storage_status: OK in evidence)
stella doctor --check check.policy.engine

# A real write path works end to end (lifecycle, exercises Postgres):
stella policy publish <args>     # or: stella policy activate <args>

# No fresh storage errors in the service log
docker logs --since 10m stellaops-policy-engine

Flag (NOT IMPLEMENTED): stella policy storage status, stella policy storage test-write, and stella policy update --test are not CLI commands. Verify storage via stella db status + the check.policy.engine storage_status evidence, and exercise a real write with a verified lifecycle subcommand (publish/activate/submit). The verified, registered stella policy subcommands (from CommandFactory.BuildPolicyCommand plus PolicyCommandGroup.AddPolicyPackCommands and PolicyInteropCommandGroup.RegisterSubcommands) are: simulate, activate, lint, edit, test, new, history, explain, init, compile, version, submit, review, publish, rollback, sign, verify-signature, lattice, verdicts, promote (core), validate-yaml, install, list-packs (pack), and export, import, validate, evaluate (interop).

Flag (corrected): findings is not a stella policy subcommand — it is a top-level command (stella findings, CommandFactory.BuildFindingsCommand). Flag (NOT IMPLEMENTED / dead code): stella policy push, pull, export-bundle, and import-bundle are defined in PolicyCommandGroup.AddDistributionCommands but that method is never called, so those four subcommands are not actually wired into the CLI. Do not rely on them for verification.

Prevention

Flag (corrected): The original draft linked Architecture: docs/modules/policy/storage.md and Database setup: docs/operations/database-configuration.md. Neither file exists. Policy storage is documented in docs/modules/policy/architecture.md; database/migration operations are covered by postgres-ops.md, ADR-004, and the migration-recovery runbook.