Runbook: Policy Engine - Evaluation Latency High

For platform on-call. Use this when Stella Ops policy evaluation is slow enough to delay release gates or CI/CD pipelines — elevated policy_run_seconds / policy_evaluation_latency_seconds P95, timeouts, or saturated workers. It walks you through diagnosing the slow phase (compilation, evaluation, caching, reachability/exception lookups) and the real tuning levers. The primary evaluator is the native C# DSL engine; OPA/Rego is an interop adapter, not the hot path.

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-003 - Policy Engine Runbooks

Reconciliation note (doc<->code): This runbook was authored against an assumed OPA/Rego-as-the-evaluator model. The Policy Engine’s primary evaluator is the native C# DSL engine (PolicyExpressionEvaluator / PolicyRuntimeEvaluationService over StellaOps.PolicyDsl); OPA/Rego is an interop adapter for teams on OPA-based toolchains, not the core hot path (see docs/modules/policy/architecture.md §13: “The C# engine remains primary; Rego serves as an interoperability adapter”). The diagnosis and resolution steps below have been corrected to match the shipped CLI surface, metric names, config keys, and Doctor check. Steps that referenced commands/config/metrics that do not exist in the codebase have been removed or flagged. Source of truth: src/Policy/**.

Metadata

FieldValue
ComponentPolicy Engine
SeverityHigh
On-call scopePlatform team
Last updated2026-05-30
Doctor checkcheck.policy.engine

The Doctor check id is check.policy.engine (source: src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Policy/Checks/PolicyEngineHealthCheck.cs). There is no check.policy.evaluation-latency check; the engine health check covers compilation, evaluation, and storage and emits a latency warning when probe evaluation exceeds 100 ms.

Symptoms

Metric names verified against src/Policy/StellaOps.Policy.Engine/Telemetry/PolicyEngineTelemetry.cs. The previously-documented metric policy_evaluation_duration_seconds does not exist and the alert name PolicyEvaluationSlow is not defined in the repo — both removed.

Impact

Impact TypeDescription
User-facingSlow release gate checks, CI/CD pipeline delays
Data integrityNo data loss; decisions are still correct and deterministic
SLA impactGate latency SLO at risk (track via policy_slo_burn_rate / policy_error_budget_remaining)

NOTE: No numeric latency SLO target (e.g. “P95 < 500ms”) is defined in source. The engine exposes SLO instruments (policy_slo_burn_rate, policy_error_budget_remaining, policy_slo_violations_total) but the thresholds are supplied by deployment-side alerting config, not the code. Set your own target during alert tuning rather than treating a fixed number as a product guarantee.

Diagnosis

Quick checks

  1. Run the Policy Engine Doctor check:

    stella doctor --check check.policy.engine

    Reports compilation_status, evaluation_status, storage_status, policy_count, evaluation_latency_p50_ms, and cache_hit_ratio. A performance_warning is emitted when probe evaluation exceeds 100 ms.

    The check requires Policy:Engine:Url (or PolicyEngine:BaseUrl) to be configured; it probes an OPA-compatible HTTP endpoint (/health, /v1/policies, /v1/data). If the engine is deployed without that URL, the check is skipped (CanRun returns false).

  2. Inspect latency / saturation metrics (Prometheus/Grafana, meter StellaOps.Policy.Engine):

    • policy_run_seconds — per-run duration histogram (tags: mode, tenant, policy, outcome)
    • policy_evaluation_latency_seconds — per-batch evaluation latency (tags: tenant, policy)
    • policy_concurrent_evaluations / policy_worker_utilization — saturation gauges
    • policy_run_queue_depth — pending-run backlog per tenant
    • policy_evaluation_failures_total{reason="timeout"} — timed-out evaluations

  3. Review structured logs / traces to find the slow phase. Evaluation is traced with spans policy.select, policy.evaluate, policy.materialize, and policy.simulate (source: PolicyEngineTelemetry.Start*Activity). Logs carry traceId, policyId, version, runId, tenant, phase.

    FLAGGED (removed): the prior steps stella policy status, stella policy stats --last 10m, stella policy evaluate --profile, stella policy cache stats, stella policy analyze --complexity, and stella policy logs --filter ... do not exist in the CLI. The real stella policy subcommands are: simulate, activate, lint, edit, test, new, history, explain, init, compile, version, submit, review, publish, rollback, sign, verify-signature, lattice, verdicts, promote, plus pack/distribution commands (validate-yaml, install, list-packs, push, pull, export-bundle, import-bundle). Diagnosis therefore relies on metrics/logs/traces and the Doctor check rather than a dedicated stats/profile CLI.

Deep diagnosis

  1. Reproduce against a known input with simulate:

    stella policy simulate <policy-id> --candidate <version> --sbom <sbom-id> --explain

    --explain requests explain traces for diffed findings, which surface the rules that fired. Use --mode batch to exercise all matching SBOMs and reproduce the slow path at scale.

  2. Check compilation cost if the slow phase is policy load/compile:

    • Metrics: policy_compilation_seconds, policy_compilation_total{outcome}.
    • Compilation enforces guardrails from PolicyEngine.Compilation: MaxComplexityScore (default 750) and MaxDurationMilliseconds (default 1500). Compilations exceeding these are rejected, not merely slowed — a borderline-complex pack can spike compile time.

  3. Check the evaluation result cache (deterministic Redis + in-memory fallback; source: src/Policy/StellaOps.Policy.Engine/Caching/):

    • The Doctor check surfaces cache_hit_ratio. A low hit ratio means most evaluations recompute from scratch.
    • Cache key is pe:{policyDigest}:{subjectDigest}:{contextDigest} — changes to policy, subject, or context digests legitimately miss the cache.

  4. Check reachability / exception lookups (these run inline during evaluation and can dominate latency):

    • policy_reachability_lookup_seconds, policy_reachability_cache_hit_ratio, policy_reachability_cache_hits_total / _misses_total.
    • policy_exception_load_latency_seconds, policy_exception_application_latency_seconds, policy_exception_loaded_total.

    FLAGGED (removed): “external data fetches” diagnosis. The evaluator is forbidden from calling external HTTP except allow-listed internal services (Authority, Storage) — see architecture §7 “Side effects prohibited”. There is no general external-data fetch path to inspect; the relevant inline I/O is reachability-facts and exception loading (above).

Resolution

Immediate mitigation

  1. Raise evaluation concurrency (the real worker knob):

    // PolicyEngine config section (PolicyEngineOptions)
"PolicyEngine": {
  "Workers": {
    "MaxConcurrentEvaluations": 8   // default 4
  }
}

    Source: src/Policy/StellaOps.Policy.Engine/Options/PolicyEngineOptions.cs (PolicyEngineWorkerOptions). Restart the Policy Engine service to apply (there is no live stella policy reload / config set command — config is loaded at startup).

    FLAGGED (removed): stella policy config set opa.workers 4 and stella policy reload. No opa.workers config key, no policy config command, and no policy reload command exist.

  2. Tune the evaluation result cache. Caching is governed by PolicyEngine.EvaluationCache (PolicyEvaluationCacheOptions) and backed by Redis with an in-memory fallback. Ensure Redis is reachable and warm so the per-batch cache (pe:{policyDigest}:{subjectDigest}:{contextDigest}) is effective; verify the resulting cache_hit_ratio via the Doctor check.

    FLAGGED (removed): stella policy cache clear / cache warm and stella policy config set cache.evaluation_ttl 60s. No policy cache command and no cache.evaluation_ttl key exist; cache behaviour is configured through PolicyEngine.EvaluationCache and the Redis backend.

  3. If the bottleneck is reachability or exception loading, ensure the reachability-facts cache and exception cache are configured and warm (PolicyEngine.ReachabilityCache, PolicyEngine.ExceptionCache), and that their backing stores are healthy. Watch policy_reachability_cache_hit_ratio and policy_exception_load_latency_seconds.

Root cause fix

If a policy is too complex / compilation is slow:

  1. Reproduce locally by compiling the DSL file:

    stella policy compile <file> --optimize --format json

    --optimize enables IR optimization passes; the output reports the SHA-256 digest of the compiled IR (suppress with --no-digest). Source: CommandFactory.BuildPolicyCommand / HandlePolicyCompileAsync.

    FLAGGED (removed): stella policy compile --all. The compile command takes a single file argument; there is no --all flag. (The Doctor check’s remediation text still suggests stella policy compile --all, which is itself inaccurate — track via the Doctor plugin.)

  2. Reduce complexity below the configured guardrail. The engine rejects packs over PolicyEngine.Compilation.MaxComplexityScore (default 750) or MaxDurationMilliseconds (default 1500); split or simplify the policy so it compiles comfortably under those limits. Use stella policy lint <file> to surface structural issues.

    FLAGGED (removed): stella policy analyze --suggest-optimizations and stella policy refactor --auto-split. Neither analyze nor refactor exists as a stella policy subcommand.

If evaluations are timing out under load:

  1. Increase PolicyEngine.Workers.MaxConcurrentEvaluations and/or scale out Policy Engine replicas, then re-check policy_worker_utilization and policy_run_queue_depth.

  2. Confirm the result cache is healthy (above) so repeat evaluations of unchanged subjects are served from cache rather than recomputed.

DRAFT / roadmap (not implemented): “Enable OPA partial evaluation” and “pre-compile policies” were carried over from the OPA-evaluator framing. The shipped primary engine is the native C# DSL evaluator, which has no OPA partial_eval toggle (opa.partial_eval is not a config key in the repo). The EmbeddedOpaEvaluator (StellaOps.Policy.Interop) shells out to a bundled opa binary only to evaluate external Rego artifacts for interop; it is not on the gate hot path. If/when OPA-side performance tuning becomes a supported lever, document the concrete config here.

Verification

# Re-run the engine health check (latency + cache surfaced in evidence)
stella doctor --check check.policy.engine

# Reproduce the evaluation and confirm it completes within budget
stella policy simulate <policy-id> --candidate <version> --sbom <sbom-id> --explain

Then confirm in metrics that policy_run_seconds / policy_evaluation_latency_seconds P95 has recovered, policy_evaluation_failures_total{reason="timeout"} has stopped incrementing, and cache_hit_ratio is back to expected levels.

Prevention