Runbook: Policy Engine - OPA Evaluation Failure

For platform on-call. Use this when an OPA-backed evaluation fails in Stella Ops — a Rego attestation gate erroring out, or embedded-OPA Rego interop returning a subprocess failure. Read the reconciliation note first: Stella’s Policy Engine is not an OPA deployment. OPA appears only in two narrow, optional, currently-unwired surfaces (an embedded opa subprocess for Rego interop, and an external attestation-gate adapter). Native DSL/IR policy evaluation — the default path — is unaffected by any OPA outage. If neither OPA surface appears in your logs, this is the wrong runbook: see Evaluation latency high or Compilation errors.

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-003 - Policy Engine Runbooks

Reconciliation note (verify against src/Policy): The original draft of this runbook assumed the StellaOps Policy Engine is an Open Policy Agent (OPA) deployment whose long-running opa process can crash. That is not how the engine works. The core Policy Engine (src/Policy/StellaOps.Policy.Engine) is a first-party .NET service with its own DSL (StellaOps.PolicyDsl), IR compiler, and deterministic evaluator (PolicyRuntimeEvaluator). It does not embed or manage a persistent OPA server, and no OPA container/sidecar ships in devops/compose. OPA appears only in two narrow, optional roles:

  1. Embedded OPA binary (offline Rego interop) — StellaOps.Policy.Interop.Evaluation.EmbeddedOpaEvaluator shells out to a bundled opa/opa.exe binary as a short-lived subprocess for Rego bundle/source evaluation and opa check syntax validation. It is not a daemon; it auto-discovers the binary and falls back gracefully when the binary is absent. A “crash” here is a non-zero subprocess exit code or a 30s timeout, not a downed service.

    Verified (dormant surface): EmbeddedOpaEvaluator / IEmbeddedOpaEvaluator are referenced only inside the StellaOps.Policy.Interop library (the class, its interface, and a DI stub). Its DI registrar AddEmbeddedOpaEvaluator is an empty stub (“Implementation registered in TASK-05” — body just return services;), so nothing registers or resolves it, and no shipping CLI command or service invokes it. The stella policy validate / evaluate / import commands do not route through it (see below). Treat this surface as present-but-unwired today; a subprocess “crash” only manifests if custom code constructs EmbeddedOpaEvaluator directly.

  2. External OPA gate (attestation verification) — StellaOps.Policy.Gates.Opa.OpaGateAdapter + HttpOpaClient can call an operator-provided external OPA HTTP server (default http://localhost:8181) to evaluate a Rego attestation gate. Stella does not ship or supervise this server.

    Verified (library capability, not wired): OpaGateAdapter, HttpOpaClient, OpaClientOptions, and OpaGateOptions are referenced only within their own definition files in StellaOps.Policy (plus that library’s tests). There is no DI registration, no OpaClientOptions/OpaGateOptions configuration binding, and no gate-registry wiring anywhere in src/ outside tests. The gate is a building block that requires custom operator/integrator wiring; it is not on any release-blocking path by default.

This runbook has been rewritten to reflect those two real surfaces. Commands, metrics, alerts, and check IDs that do not exist in src/ are flagged inline as NOT IMPLEMENTED. Where the original draft attributed CLI commands to components they do not actually call, the attribution is corrected inline.

Metadata

FieldValue
ComponentPolicy Engine (OPA interop / attestation gate)
SeverityHigh (Critical only when an OPA attestation gate is on the release-blocking path)
On-call scopePlatform team
Last updated2026-01-17 (reconciled against src/Policy)
Doctor checkcheck.policy.engine (source: src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Policy/Checks/PolicyEngineHealthCheck.cs)

Flag (corrected): The original check.policy.opa-health Doctor check does not exist. The Policy Engine plugin registers exactly one check, check.policy.engine, which probes the configured policy-engine URL (Policy:Engine:Url or PolicyEngine:BaseUrl, default http://localhost:8181) using OPA-style endpoints (/health, /v1/policies, /v1/data, /v1/status) and records engine_type in its evidence.

Symptoms

Flag (NOT IMPLEMENTED): The original symptoms referenced an alert PolicyOPACrashed and a metric policy_opa_restarts_total. Neither exists anywhere in src/. Do not page on them. If you need restart/health alerting for an external OPA server, it must be defined against that external deployment’s own metrics, not against a Stella-emitted series.

Impact

Impact TypeDescription
User-facingOnly flows that depend on OPA interop are affected, and only if an operator has explicitly wired them (neither surface ships wired — see the reconciliation note): (a) custom code that constructs EmbeddedOpaEvaluator for offline Rego evaluation, and (b) any release gate an integrator has wired through OpaGateAdapter. The stock CLI interop commands (stella policy validate / evaluate / import) do not touch OPA — they use the native JSON importer and native gate evaluation. Native DSL/IR policy evaluation (the default path) is unaffected by an OPA outage.
Data integrityNo data loss. Subprocess and HTTP-gate failures are surfaced as errors/denials; nothing is persisted incorrectly.
SLA impactOnly relevant where an integrator has wired an OPA attestation gate. If such a gate sits on the release-blocking path with FailOnError = true (the default for OpaGateOptions), releases through that gate block until OPA recovers. With FailOnError = false the gate passes open (evidence quality degrades but the pipeline proceeds).

Diagnosis

Quick checks

  1. Check Doctor diagnostics (verified check ID):

    stella doctor --check check.policy.engine

    Look for: engine_url, engine_type, connection_error_type, compilation_status / evaluation_status / storage_status in the evidence.

  2. Confirm which OPA surface is involved. Read the relevant logs and identify the failing component:

    • EmbeddedOpaEvaluator → offline Rego subprocess, only from custom code that constructs it (not reachable from the stock CLI).
    • OpaGateAdapter / HttpOpaClient → external OPA HTTP server, only where an integrator has wired the gate. These have different fixes; do not conflate them. If neither component appears in logs, the failure is almost certainly in native DSL/IR evaluation, not OPA — use the sibling runbooks (policy-evaluation-slow.md, policy-compilation-failed.md).

Flag (NOT IMPLEMENTED): stella policy status and stella policy opa logs (used by the original draft) are not CLI commands. Verified stella policy subcommands are: simulate, activate, lint, edit, test, new, history, explain, init, compile, version, submit, review, publish, rollback, sign, verify-signature, lattice, verdicts, promote, validate-yaml, install, list-packs, push, pull, export-bundle, import-bundle, export, import, validate, evaluate. (Confirmed against CommandFactory.BuildPolicyCommand, PolicyCommandGroup, and PolicyInteropCommandGroup.) For service logs, use your container/host log tooling (e.g. docker logs, journalctl) against the policy-engine container.

Deep diagnosis (external OPA gate)

  1. Reach the external OPA server directly at the configured URL (OpaClientOptions.BaseUrl, default http://localhost:8181):

    curl -s http://localhost:8181/health

    HttpOpaClient.HealthCheckAsync performs exactly this GET /health.

  2. Inspect the gate configuration (OpaGateOptions): PolicyPath (default stella/policy/allow), FailOnError, TrustedKeyIds. A FailOnError = true gate is the one that blocks releases on OPA outage.

  3. Check the request timeout. OpaClientOptions.TimeoutSeconds defaults to 10s; HttpOpaClient returns “Request timed out after N seconds” when exceeded.

Deep diagnosis (embedded OPA binary)

Scope note (corrected): EmbeddedOpaEvaluator is not invoked by any shipping CLI command or service (its DI registrar is an empty stub). This section therefore applies only to custom code that constructs EmbeddedOpaEvaluator directly. The stock stella policy validate / evaluate / import commands do not exercise it, so they cannot be used to reproduce an embedded-OPA failure.

  1. Verify the binary is discoverable. EmbeddedOpaEvaluator.DiscoverOpaBinary searches, in order: <app>/tools/opa[.exe], <app>/opa[.exe], ~/.stella/tools/opa[.exe], then each PATH entry (the first existing file wins). IsAvailableAsync returns false when none resolves, and the evaluator returns “OPA binary not found.” for every operation.

  2. Reproduce in the calling code, not via the CLI. When EmbeddedOpaEvaluator is invoked directly, a non-zero subprocess exit surfaces as “OPA evaluation failed (exit N):” (or “OPA bundle evaluation failed (exit N):” for EvaluateBundleAsync), carrying the raw OPA stderr; a TimeoutException after the 30s default indicates a runaway/large evaluation. The underlying commands it runs are opa eval --data … --input … <query> --format json, opa eval --bundle … --input … <query> --format json, and opa check <file> --format json.

Flag (NOT IMPLEMENTED): stella policy stats --opa-metrics, stella policy list --last-error, stella policy opa config show, and stella policy analyze --detect-loops do not exist. The 30s subprocess timeout in EmbeddedOpaEvaluator (the constructor default; configurable only via its timeout constructor argument) is the only “infinite loop” guard; there is no loop-detection command.

Resolution

Immediate mitigation

External OPA gate down (release-blocking path):

  1. Recover the external OPA server (restart it via whatever supervises it — it is operator-managed, not a Stella container). Stella does not expose a stella policy opa restart command (NOT IMPLEMENTED).

  2. If the gate must not block while OPA is down, set the gate’s FailOnError = false so OpaGateAdapter passes open on OPA errors, then redeploy/reload the Policy Engine configuration through your normal config path.

    Warning: Passing an attestation gate open degrades evidence quality. Only do this if compliance allows it, and re-enable FailOnError = true once OPA recovers.

Embedded OPA binary missing/failing:

  1. Provide the binary at one of the discovered locations (e.g. <app>/tools/opa[.exe] or on PATH). The interop path falls back gracefully when absent — native DSL/IR evaluation continues to work without it, so this only blocks Rego import/validate/evaluate.

Flag (NOT IMPLEMENTED): stella policy opa start --safe-mode, stella policy config set failopen true, and stella policy reload do not exist. Fail-open behaviour is the per-gate OpaGateOptions.FailOnError flag, not a global failopen config key.

Root cause fix

If the external OPA server was OOM-killed or overloaded:

  1. Size and tune the external OPA deployment per its own operational docs (OPA memory limits and GC are configured on the OPA process, not via Stella). Stella exposes none of stella policy opa config set memory_limit|gc_* (NOT IMPLEMENTED).

  2. Reduce gate evaluation cost: trim the input payload built by OpaGateAdapter.BuildOpaInput (merge result, context, supply-chain evidence) and the Rego at OpaGateOptions.PolicyPath.

If a Rego policy itself is faulty (interop path):

  1. Validate the policy with the verified interop command (note: this uses --file, not a positional argument):

    stella policy validate --file <policy.rego>

    Flag (corrected): stella policy validate does not run opa check and does not call EmbeddedOpaEvaluator.ValidateSyntaxAsync. It runs JsonPolicyImporter.ImportFromStringAsync with ValidateOnly = true (structural validation against the PolicyPack v2 schema, with format auto-detection via FormatDetector). For a .rego file it validates the importer’s mapping of the Rego, not the Rego’s own syntax via OPA. If you need genuine opa check syntax validation, run the opa binary directly (opa check <file>).

  2. Fix the policy, then re-validate and re-import (both use --file):

    stella policy validate --file <fixed-policy.rego>
stella policy import --file <fixed-policy.rego>   # JsonPolicyImporter (see note)

    Flag (corrected): stella policy import takes --file/-f (required), not a positional path. It also does not use RegoPolicyImporter: the command constructs a JsonPolicyImporter for both the JSON and Rego branches and reports the mapping. (RegoPolicyImporter exists as a library class but is not the type the import command instantiates.) Use --validate-only or --dry-run to avoid persisting.

Flag (NOT IMPLEMENTED): stella policy list --status error, stella policy disable/enable/update <policy-id> are not CLI commands. Policy lifecycle (draft → submit → review → publish → activate → rollback) is managed through the verified stella policy submit|review|publish|activate|rollback subcommands, not disable/enable.

If a policy-pack bundle failed to load (interop path):

  1. Re-export/re-import the offline bundle with the verified commands:
    stella policy export-bundle --policy <file> --output bundle.tar.gz
stella policy import-bundle --bundle bundle.tar.gz --verify

    Caution (corrected): these commands operate on Stella policy-pack bundles — a gzip stream of FILE:<path>:<len> records plus an index.json manifest (ExportBundleAsync / ImportBundleAsync). This is not an OPA bundle. The unrelated EmbeddedOpaEvaluator.EvaluateBundleAsync (opa eval --bundle …) consumes an OPA bundle.tar.gz and is not invoked by any CLI command. Do not feed an OPA bundle to import-bundle or vice versa.

Flag (NOT IMPLEMENTED): stella policy bundle verify|build|load are not CLI commands; the verified bundle commands are export-bundle / import-bundle (and the OCI push / pull).

If the configuration is wrong:

  1. Verify the engine/gate URL Stella is actually using:
    • Doctor reads Policy:Engine:Url or PolicyEngine:BaseUrl (default http://localhost:8181).
    • The gate client reads OpaClientOptions.BaseUrl (section Opa, default http://localhost:8181). Correct these in the Policy Engine configuration and restart the service via your normal deployment path.

Flag (NOT IMPLEMENTED): stella policy opa config reset and stella policy opa config set workers|decision_log ... do not exist.

Verification

# Re-run the verified Doctor check
stella doctor --check check.policy.engine

# (External gate) confirm the OPA server answers its health endpoint
curl -s http://localhost:8181/health

# (Interop) confirm policy validate/evaluate now succeed (note option syntax;
# these run the native importer + native gate evaluation, NOT OPA)
stella policy validate --file <policy.rego>
stella policy evaluate --policy <policy-file> --input <evidence.json>

# Confirm native DSL/IR evaluation is unaffected (independent of OPA)
stella policy test <policy-dsl-file>

Flag (corrected / NOT IMPLEMENTED): stella policy validate and stella policy evaluate use --file / --policy + --input options (not positional arguments) and exercise the native JSON importer and native EvaluateGate logic — they are not OPA health probes. stella policy status, stella policy opa health, stella policy evaluate --test, and stella policy stats --watch are not CLI commands. Use check.policy.engine for engine health and stella policy test for DSL fixture coverage.

Prevention