Runbook: Policy Engine — Policy Version Conflicts

Use this runbook when policy-gated release decisions look inconsistent or stale — for example, two policy-engine replicas reaching different verdicts for the same artifact, or a publish/activate that does not appear to take effect. It is written for the Platform on-call team and assumes familiarity with the Stella Ops policy versioning and activation model (summarized below).

The key thing to internalize: Stella Ops policy packs are versioned in PostgreSQL, not distributed as polled OPA bundles. “Different nodes on different versions” is therefore almost always a stale process cache at a policy-engine replica, not a distribution-layer failure. The corrected version model is documented under Version model (ground truth).

Metadata

Field	Value
Component	Policy Engine
Severity	Medium
On-call scope	Platform team
Last updated	2026-05-30
Doctor check	`check.policy.engine` (plugin `stellaops.doctor.policy`; `PolicyEngineHealthCheck`)

Source-of-truth note (reconciled 2026-05-30): there is no dedicated check.policy.version-consistency Doctor check and no PolicyVersionMismatch alert in the codebase. The closest implemented diagnostic is check.policy.engine, which probes the policy engine’s compilation, evaluation, and storage health (it reports a policy_count evidence field but does not compare versions across replicas — and despite its name it surfaces no policy_versions evidence field: the storage probe counts stored data entries internally but never emits that count). The version model is also not the multi-node “bundle distribution” model this runbook originally assumed — see the corrected model below. Items that describe an unimplemented design are flagged NOT IMPLEMENTED.

Version model (ground truth)

Policy packs are versioned in PostgreSQL, not distributed as polled OPA bundles:

Each pack has an append-only, immutable set of policy.pack_versions rows (sequential integer version, rules_hash, is_published, published_at, published_by). Published versions cannot be mutated, unpublished, deleted, or renumbered (PackVersionEntity, PolicyVersioningImmutabilityTests).
“Which version is live” is a single active_version column on policy.packs, scoped per tenant. Activation (PackRepository.SetActiveVersionAsync) only succeeds when the target version exists and is_published = TRUE. Activation never alters version content.
Policy-engine replicas read the active version from the shared Postgres schema; there is no per-node bundle cache, no --all-nodes sync fan-out, and no OPA bundle-polling interval to tune. “Different nodes on different versions” is therefore almost always stale connection / caching at the engine process, not a distribution-layer divergence.
OPA, where used, is consumed as a gate adapter over OPA’s REST API (OpaGateAdapter → HttpOpaClient). The gate adapter’s OPA base URL comes from the Opa config section (OpaClientOptions.BaseUrl, SectionName = "Opa"), default http://localhost:8181. (Distinct from the Doctor health check below, which reads Policy:Engine:Url / PolicyEngine:BaseUrl — same default URL, but a different config key.) OPA is not a bundle server that StellaOps polls.

Symptoms

[ ] Policy evaluation returning unexpected results
[ ] check.policy.engine reporting Fail/Warn for compilation, evaluation, or storage
[ ] Error from activation: target version not published, or pack/version not found
[ ] A policy-engine replica evaluating against a stale active version after a publish/activate
[ ] Inconsistent gate decisions for the same artifact across replicas

Impact

Impact Type	Description
User-facing	Inconsistent policy decisions; unpredictable gate results
Data integrity	Decisions may not match expected policy behavior
SLA impact	Gate accuracy SLO violated; trust in decisions reduced

Diagnosis

Quick checks

Check Doctor diagnostics (the implemented policy-engine health check):
```
stella doctor --check check.policy.engine
```
This probes compilation, evaluation, and storage against the policy engine URL (Policy:Engine:Url / PolicyEngine:BaseUrl). Its evidence includes engine_type, engine_version, engine_url, the *_status fields, policy_count, compilation_time_ms, evaluation_latency_p50_ms, and cache_hit_ratio (there is no policy_versions evidence field).
Inspect a policy’s run history (implemented). Note: history lists policy run records (filterable by --from / --to / --status), not pack version/publish state — to confirm which versions are published, query policy.pack_versions directly (see Deep diagnosis):
```
stella policy history <policy-id>
```

Check the engine and its dependencies (implemented Doctor checks):

stella doctor --check check.postgres.connectivity
stella doctor --check check.policy.engine

NOT IMPLEMENTED. stella policy version --all-nodes, stella policy active --show-version, stella policy version diff --all-instances, stella policy bundle status, stella policy deployments list, and stella policy opa bundle-status do not exist. The stella policy command group in the CLI exposes lint, compile, simulate, version bump, submit, review, publish, activate, rollback, promote, history, explain, edit, test, new, init, sign, verify-signature, lattice, and verdicts — there is no --all-nodes flag, no bundle/deployment/opa subcommands, and no multi-node version-diff surface (the version model is single shared Postgres, not per-node bundles).

Deep diagnosis

Confirm the active version per tenant in the database. The live version is the active_version column of policy.packs for the affected tenant; published versions live in policy.pack_versions. A replica diverging from this row is the real “version conflict” condition.
Confirm storage reachability. check.policy.engine storage probe failing, or check.postgres.connectivity failing, points at the shared store the engine reads versions from.
Check OPA reachability (if OPA gates are in use). Two different consumers reach OPA over REST: the OpaGateAdapter (base URL from the Opa config section, default http://localhost:8181) and the check.policy.engine Doctor probe (URL from Policy:Engine:Url / PolicyEngine:BaseUrl, same default). A failing check.policy.engine evaluation/compilation probe surfaces OPA REST errors and records them as evaluation_error / last_compilation_error evidence.

Resolution

Immediate mitigation

Re-pin the active version explicitly (implemented). Activate the intended, already-published version for the affected tenant; only published versions can be activated, and activation does not mutate version content:
```
stella policy activate <policy-id> --version <version>
```
Roll back to a previous published version if the newly-activated one is the problem (implemented). The flag is --target-version (defaults to the previous version when omitted):
```
stella policy rollback <policy-id> --target-version <version>
```
Restart the diverging policy-engine replica so it re-reads the active version from Postgres. Use your deployment tooling (e.g. docker compose restart policy-engine); there is no stella service restart CLI command.

Root cause fix

If a replica is reading a stale version:

Confirm the authoritative active version in policy.packs.active_version for the tenant (see Deep diagnosis).
Restart / cycle the replica with the deployment tooling so it reloads from the shared store. Because all replicas read the same Postgres row, divergence is a process-cache condition, not a distribution failure.

If storage is unavailable:

Run stella doctor --check check.postgres.connectivity and stella doctor --check check.policy.engine.
Follow policy-storage-unavailable.md to restore the policy store, then re-verify the active version.

If an OPA gate is misbehaving:

Confirm OPA is reachable. The gate adapter uses the Opa config section (default http://localhost:8181); the Doctor probe uses Policy:Engine:Url / PolicyEngine:BaseUrl (same default). A failing check.policy.engine evaluation/compilation probe surfaces the OPA REST error.
If OPA itself has crashed, follow policy-opa-crash.md.

NOT IMPLEMENTED. The original “bundle distribution”, “node resync”, “deployment locking”, and “OPA bundle polling” remediations describe a design that does not exist in code. There is no stella policy sync, stella policy pin, stella policy bundle …, stella policy status --node, stella policy resolve-conflict, stella policy config set deployment.locking, or stella policy opa config set bundle.polling.* (the bundle.polling.min_delay_seconds / max_delay_seconds config keys appear nowhere in the source). Treat version convergence as: activate the published version + restart the stale replica.

Verification

# Re-run the policy engine health check (compilation/evaluation/storage)
stella doctor --check check.policy.engine

# Review the policy's run history (NB: run records, not version/publish state;
# to confirm a version is published, query policy.pack_versions directly)
stella policy history <policy-id>

# Re-run a DSL simulation against the policy to confirm expected output
# (this is the policy-DSL `simulate`: a .stella file + a signals JSON context)
stella policy simulate <policy-file> --signals <signals.json>

The authoritative cross-replica check is confirming each engine evaluates against the same policy.packs.active_version row; there is no stella policy version --all-nodes, stella policy evaluate --all-nodes, or stella policy logs command to do this from the CLI today.

Prevention

[ ] Immutability: Rely on the append-only version model — published versions cannot be mutated, so a bad version is corrected by activating/rolling back to a good published one, never by editing in place.
[ ] Promotion gates: Promote a policy through environments (stella policy promote) and review/approve (stella policy review, stella policy submit) before activating in production.
[ ] Monitoring: Schedule check.policy.engine so engine compilation/evaluation/ storage health (and policy_count / cache_hit_ratio) is observed continuously.
[ ] Testing: Simulate (stella policy simulate) and lint (stella policy lint) a policy before publishing/activating.

NOT IMPLEMENTED. “Enable deployment locking” and “configure aggressive bundle polling for fast convergence” describe controls that do not exist (no deployment-lock config, no bundle-polling layer). They are retained here only to mark them as absent.

Architecture: docs/modules/policy/architecture.md (Policy module dossier; versioning/activation model). There is no docs/modules/policy/versioning.md.
Related runbooks: policy-opa-crash.md, policy-storage-unavailable.md, policy-compilation-failed.md, policy-evaluation-slow.md, policy-incident.md (all present in this directory).
Doctor check source: src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Policy/Checks/PolicyEngineHealthCheck.cs
Version model source: src/Policy/__Libraries/StellaOps.Policy.Persistence/Postgres/ (Models/PackVersionEntity.cs, Repositories/PackRepository.cs, Repositories/PackVersionRepository.cs).

The originally-referenced docs/operations/policy-deployment.md does not exist in the docs tree and has been dropped.