Policy Incident Runbook

The on-call response procedure for a Policy Engine incident in Stella Ops — a failed or incorrect policy evaluation, a bad policy publication, or a blocked promotion traced to policy. It gives platform on-call a deterministic triage → mitigation → evidence flow that keeps every action attributable and replayable. For deeper, failure-specific detail (DSL compilation errors, evaluation latency, storage/OPA outages), branch to the focused runbooks under Related runbooks.

Supported Scope

This runbook covers policy incident handling for the supported Stella Ops deployment model: Docker Compose, Offline Kit, signed release manifests, and host/service-manager operations.

Kubernetes and Helm are unsupported. Do not use historical Helm mock overlays, chart rendering, Kubernetes canaries, kubectl logs, or pod/deployment scaling as Stella Ops incident-response steps.

Triage

1. Capture the incident id, tenant, environment, policy bundle version, release manifest digest, and affected service.
2. Export the current policy bundle and policy evaluation evidence from the Stella Ops API or persisted evidence store.
3. Verify the policy bundle digest against the signed release manifest or approved policy publication event.
4. Check policy service health through the supported service endpoint and host logs.
5. If policy compilation failed, freeze further promotions for the affected environment until the bundle is corrected and republished.

Mitigation

1. Revert to the last signed policy bundle only through the approved policy publication workflow.
2. Restart affected services through the supported Compose or host service-manager procedure.
3. Re-run the policy evaluation that failed and attach the before/after evidence to the incident.
4. Record operator approval, bundle digest, release manifest digest, and rollback decision.

Evidence

• Incident id and timestamps.
• Policy bundle id and SHA-256 before and after mitigation.
• Release manifest digest.
• Service health output.
• Relevant application logs from the supported deployment host.
• Promotion freeze/unfreeze decision.

Related runbooks

Use this runbook for incident coordination, and branch to the focused runbooks for the specific failure mode:

• Policy compilation errors — DSL compile/lint diagnostics, failed version activation.
• Policy evaluation latency high — slow gate decisions, timeouts, cache/worker tuning.
• Policy storage backend down — PostgreSQL on the Policy DSN unreachable.
• OPA evaluation failure — optional Rego interop / attestation-gate surface.
• Policy version mismatch — divergent active versions across environments.