Policy Test Fixtures (stella policy test)

Audience: policy authors and CI engineers writing coverage fixtures for stella-dsl@1 policies. Companion to the Policy DSL guide (§9.4 coverage fixtures) and the Policy Lifecycle guide.

stella policy test <file> compiles a policy, evaluates every fixture in a fixtures directory through the real DSL evaluator, and compares the resulting verdict against the fixture’s expectations. Every reported pass is backed by an evaluation — the command never reports a pass for a fixture it did not evaluate, and a run that evaluates nothing exits non-zero.

stella policy test ./policies/default.stella \
  --fixtures ./tests/policy/default/cases \
  --format json --output coverage.json

Fixtures default to tests/policy/<policy-name>/cases/**/*.json (relative to the policy file, then to the working directory). --filter <text> keeps only fixtures whose file name contains <text>.

1 · Fixture file format

One JSON object per file. Property names are snake_case and are matched exactly.

{
  "description": "vendor cleared the component",
  "expected_outcome": "pass",
  "input": {
    "vex": { "status": "not_affected" },
    "finding": { "severity": "critical" }
  },
  "expected_findings": [
    { "status": "not_affected", "rule": "vex_clears" }
  ]
}

A fixture that is not valid JSON, declares no expected_outcome, has no input, or whose input is not a JSON object is reported as error — never as a pass and never as a silent skip — and the run exits non-zero.

2 · input — building the evaluation context

Each top-level property of input becomes a signal available to the policy under that name, so "finding": { "severity": "critical" } makes finding.severity resolvable. JSON maps to policy values as: object → nested map, array → list (usable with in), string → string, number → decimal, true/false → boolean, null → null.

Signal names follow the DSL evaluation context (see dsl.md §5): sbom, advisory, vex, env, telemetry, signals, secret. Referencing a signal the fixture does not provide yields null, which is falsey — the same as in the Policy Engine service.

3 · expected_outcome

expected_outcome is matched case-insensitively and accepts three shapes:

1. pass / fail— asserts the release-gate outcome derived from the verdict status:

   Verdict status	Gate outcome
   affected, escalated	fail
   not_affected, fixed, suppressed, ignored, deferred, under_investigation, warned	pass

   If a policy assigns a status outside this vocabulary, pass/fail has no defined meaning for it: the fixture is reported as an error telling you to assert the status directly, rather than the runner guessing.

2. no_match— asserts that no rule matched at all.

3. any other value — asserts the verdict status verbatim (case-insensitive), e.g. "expected_outcome": "under_investigation".

4 · expected_findings

Optional. A policy evaluation produces exactly one verdict, so expected_findings holds at most one object; declaring more is a fixture defect and is reported as an error rather than silently comparing only the first. Each property of that object is asserted against the verdict:

5 · Evaluation semantics

The fixture runner uses the StellaOps.PolicyDsl evaluator and mirrors the Policy Engine service’s rule semantics:

• Rules run in ascending priority; ties are broken by declaration order (dsl.md §7.1).
• The first matching rule wins and only its then actions are applied. Evaluation stops there.
• else branches are not applied — the service evaluator returns on the first match and never runs them.
• A matching rule that assigns no status yields affected, and no matching rule at all also yields affected: evaluation is fail-closed, so an unhandled finding stays actionable.
• status := X sets the status, severity := X the severity, any other assignment target and every annotate becomes an annotation. ignore → ignored, defer → deferred, warn → records the warning and (if no status was set) warned, escalate to X sets severity, requireVex → affected when every condition holds, otherwise suppressed.
• Evaluation is deterministic and offline: no clock, no network, no ambient state. The same fixture and policy produce the same verdict on every machine, and the report carries the policy SHA-256 checksum so a result is attributable to an exact compiled policy.

6 · Known limitations

Built-in functions and indexers are not evaluated here. vex.any(...), normalize_cvss(...), exists(...), secret.hasFinding(), severity_band(...), sbom.any_component(...) and index expressions are implemented by the Policy Engine service evaluator, which lives inside the policy-engine service and is not reachable from the CLI. Rather than evaluate such a construct as false — which would produce a confident but wrong verdict — stella policy test refuses the whole run, names each unsupported construct with its rule and source location, and exits 4. Policies restricted to comparisons (==, !=, <, <=, >, >=), membership (in, not in), boolean operators and field access evaluate fully.

Comparison semantics match the service evaluator, including severity words: critical > high > medium/moderate > low > informational/info > none > unknown, so finding.severity >= "high" behaves as an author expects.

Exceptions, unknown budgets, confidence scoring and Evidence-Weighted Score enrichment are applied by the Policy Engine service after rule evaluation and are outside the scope of fixture tests. A green fixture run asserts rule behaviour, not the full production verdict.

7 · Exit codes

--fail-fast stops after the first non-passing fixture. Exit codes always reflect the aggregate truth of the run: a run with zero evaluated fixtures never exits 0.