VEX Consensus Operations Runbook

Audience: Operators who run the VEX decision path and configure issuer trust. Verified against src/: 2026-10-04 (src/Vulnerabilities, src/Policy, src/Scanner/StellaOps.Scanner.Worker, src/Findings, src/Cli).

This runbook describes how Stella Ops turns VEX and advisory statements into one effective status per issue, where that status is applied, and how to configure and check it. For symptom-driven diagnosis, use the VEX Consensus Troubleshooting Guide.

Earlier model. This page replaces the “trust lattice” runbook. The product does not run a Provenance/Coverage/Replayability (PCR) trust vector, freshness decay, claim-strength multipliers, a conflict penalty, confidence-threshold VEX gates, or the VexLens and Excititor services. No service reads the trust-lattice, policy-gates or excititor-calibration sample files under devops/etc/. The retired design is archived at docs-archive/modules/excititor/retired-20260926/trust-lattice.md.

1. Terms

2. Data flow

1. Ingest. The Vulnerabilities hub stores statements from its source connectors (advisory feeds, distro trackers, Red Hat and SUSE CSAF) and from signed OpenVEX pushes (POST /api/vulnerabilities/v1/issuers/{issuerId}/vex, section 5.4). A change to an issue’s statements produces a consensus.changed event on the hub stream vuln.consensus.
2. Read. The Scanner Worker, Policy’s gate projection and the CLI read an issue’s live statements from the same hub route, GET /api/vulnerabilities/v1/issues/{vulnerabilityId}/{productKey}/facts (scope advisory:read). Policy’s first-fill seed reads the same statements in bulk from the hub corpus section consensus-inputs.
3. Evaluate. Each of these readers evaluates the statements with the same function, ConsensusEvaluator (src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Consensus/ConsensusEvaluator.cs).
4. Apply. The Scanner Worker, the release findings lookup, the Findings read model and the CLI use the result. The drift gate reads the VEX status that a knowledge snapshot records (section 4).

3. Consensus rules

ConsensusEvaluator.Evaluate is a pure function of the statements, the trust configuration and the tenant overlays:

1. An issue with no statements evaluates to under_investigation. The absence of statements is never not_affected. A tenant status overlay still decides such an issue for that tenant (rule 7).
2. Each issuer has a tier: the IssuerTiers entry for the issuer, else DefaultTier (community). Tiers rank by TierPrecedence, highest first: vendor, distro, aggregator, community. A tier name that is not in that list ranks below every listed tier.
3. Only the statements of the best-ranked tier present count. A lower tier never overrides a higher tier.
4. When equally trusted issuers disagree on concrete statuses, the result is under_investigation with issuer_confirmation_required. The user must confirm an issuer. Claims from one issuer still use status precedence (affected, fixed, not_affected, under_investigation); an open investigation does not contradict another issuer’s concrete claim. Other status values rank last.
5. The winning issuers are the issuers of that tier that state the winning status. The justification is the first non-empty justification among their statements.
6. contested is true when any statement, in any tier, states a different status.
7. A tenant status overlay replaces the outcome for that tenant; between two live status overlays on one issue, the later assertion wins. A tenant trust overlay assigns an issuer a different tier for that tenant. overlayApplied is true when an overlay changed the outcome. A live case-untrust overlay takes precedence over status assertions and holds the issue at under_investigation until that hold is withdrawn. Its optional issuer is audit context.
8. The effective affected range is the set of ranges that the winning statements carry.
9. The decision hash is the SHA-256 of the canonical form of the statements, the trust configuration, the overlays and the outcome. The same arguments always give the same hash.

Default install. The shared defaults rank the known distribution feeds above general aggregators; unlisted issuers retain community. Exact defaults and the supported operator configuration are in section 5.2.

4. Where the effective status is applied

5. Configuration

5.1 Gate projection and release lookup

5.2 Issuer tiers

Policy and Scanner use the shared Vulnerabilities:Consensus:Trust section and ConsensusTrustOptions defaults. The shipped exact issuer IDs debian-security-tracker, alpine-secdb, redhat-csaf and suse-csaf rank distro; osv and nvd rank aggregator. Unlisted issuers rank community. An issuer does not gain trust from its display name or a claim that it is a vendor. Explicit vendor-tier assignments remain an operator decision.

To customize trust once, mount the same durable JSON file read-only in Policy and Scanner Worker, and set VULNERABILITIES_CONSENSUS_TRUST_FILE on both to its mounted path. A configured but missing file refuses startup. Its content has this shape:

{
  "Vulnerabilities": {
    "Consensus": {
      "Trust": {
        "IssuerTiers": { "your-verified-issuer": "vendor" }
      }
    }
  }
}

The dictionary starts with the shipped defaults; explicit entries replace the named assignments. TierPrecedence is highest first (vendor, distro, aggregator, community), and DefaultTier is community. Environment-variable settings follow the ordinary host configuration precedence. Restart both consumers after changing the shared file; new scans use the new settings.

Policy compares the canonical trust fingerprint against the one active policy.vuln_gate#issuer-trust-v3 completion receipt. A changed fingerprint re-derives existing projection rows in fenced corpus batches, without requiring a new hub event. The existing per-release walk also certifies the same trust fingerprint, so an upgrade that needs both uses one walk. A return from configuration A to B and back to A requires a new derivation of A; a historical receipt cannot certify rows now derived with B. The receipt is written only after the full walk succeeds. Existing gates keep serving their prior applied rows during the walk; a restarted walk begins again. Read actual progress and completion evidence before claiming the new configuration is fully applied.

stella vex explain uses the shared evaluator’s shipped defaults; it does not read an operator’s runtime file or tenant/environment preferences.

5.3 Drift gate

SmartDiff:Gates:BlockOnAffectedReachable (default true) controls the AffectedReachable check of the drift gate behind POST /api/v1/policy/gate/evaluate. The check blocks a promotion whose delta makes a CVE newly reachable while the target knowledge snapshot records that CVE as affected or under_investigation. Set the key to false to make the check warn instead.

5.4 Signed OpenVEX issuers

The hub accepts POST /api/vulnerabilities/v1/issuers/{issuerId}/vex only from an allowlisted issuer. Each entry binds one issuer to one OAuth client and one public key:

The compose stack and the release bundle allowlist no issuer. Until an operator adds one, every push answers 403 issuer_not_authorized. The request body is a DSSE envelope that carries an OpenVEX document; the hub rejects it unless the signature verifies.

6. Tenant VEX decisions and trust overrides

The console records VEX decisions (/api/findings/v1/vex-decisions) in Findings, and Policy’s gate projection applies them:

1. Each decision write appends the decision’s overlay state to the tenant’s findings.vex-decisions feed in the same transaction.
2. The projection host pulls the feed for each tenant that has a policy pack or an overlay and for each tenant Policy’s tenants replica lists as active, writes the state to policy.vuln_tenant_overlay and derives the decided issue again. A tenant without a decision reads the installation’s system default overlay, if one is configured. The release findings lookup and the Findings read model then use the decision; overlayApplied is true when it changed the outcome.

A decision with a validity window applies only inside the window; the projection derives the issue again when the window opens and when it closes. Between two live decisions on one issue, the later one wins. The drift gate does not read decisions: it reads the status that the target knowledge snapshot records.

The policy-engine Authority client needs findings:projection:read. On an upgraded estate, restart Authority (or reconcile the authority.clients row) so that the grant takes effect.

Case trust overrides (/api/findings/v1/ledger/vex-trust-overrides) use the same feed. trusted: false holds (tenant, vulnerability, product key) at under_investigation and prevents a status decision from clearing that issue. A component purl is mapped to the hub’s versionless package key, so the hold is tenant-wide across package versions. vexSource is audit context; it does not narrow this case hold or re-tier an issuer. Setting trusted: true or deleting a row withdraws that row’s hold. Other live holds on the issue still apply.

The v3 feed snapshots decisions, case-trust rows and remembered issuer preferences, ordered by canonical overlay identity. Policy’s policy.vuln_gate.vex_overlays_v3 consumer starts from a snapshot rather than adopting an earlier consumer’s checkpoint. Upgrade Findings before Policy so that the consumer receives the v3 payload. The shared consensus decision-hash format is v3.

Equal-tier issuer disagreements require confirmation. A case answer uses the existing tenant/vulnerability/product overlay. Remembering an issuer is a separate console action: Findings stores that preference for the selected tenant, release environment and issuer pair. Editing or withdrawing a remembered preference re-derives its affected environment rows; effective-status reads fail closed while that environment’s derivation is pending. See the feed contract for the preference routes, identities and withdrawal behavior.

Feed contract: VEX decision overlay feed.

7. Routine checks

1. Confirm that the projection runs. The policy-engine log shows policy-engine starting the gate projection loop at start and Gate projection lag: <n> hub events outstanding on vuln.consensus while it drains. Expected result: the lag returns to a low value after hub ingest.
2. Spot-check one issue. Compare stella vex explain with Policy’s tenant answer (troubleshooting guide, sections 1.1 and 1.2). Expected result: the same status, unless a configured tier decides the issue or the projection has not applied a recent change yet.
3. After an issuer push, check the hub answer. Expected result: 202 Accepted with factsWritten and factsChanged.
4. After a console VEX decision, read Policy’s tenant answer for the issue (troubleshooting guide, section 1.2). Expected result: the decision’s status with overlayApplied: true after the next projection poll (PollInterval, 5 seconds by default), unless section 6 says that the decision is not applied.

7.1 Before VEX estate acceptance

Bind the running service images to one fixed accepted source ref with tools/scripts/validate/check-estate-image-staleness.sh. Check the recorded deployment chain, image digests and service health through the deployment procedure before testing VEX behavior. A clean buildinfo stamp or a healthy container alone does not prove that the current VEX producer and consumer are active.

Run tools/scripts/validate/check-vex-estate-readiness.py --self-test, then invoke that read-only checker with an explicit --repo, selected --tenant and --environment, accepted --findings-min-commit, --policy-min-commit and --scanner-min-commit (Scanner Worker), and the --expected-trust-fingerprint of the reviewed effective shared trust configuration. The optional --expected-feed-base-address must match the reviewed deployment plan; its default is http://findings.stella-ops.local:8080/. Address equality checks the plan, not network reachability.

The checker reads deployed source stamps and named metadata from the owning databases without initializing snapshots or writing decisions. It requires the v3 overlay bootstrap checkpoint to match the selected Findings producer fence, the current issuer-trust completion marker to match the supplied fingerprint, the required v3 relations to exist, and no pending derivation for the selected environment. The platform and issuer-trust markers must use the current hub epoch; their captured corpus heads may be ahead of the event drain. Exit 0 means these prerequisites were observed, 1 names an unmet prerequisite, and 2 means the observation could not be established. Every result has acceptanceVerified: false; the fingerprint is a supplied expectation, not an independently reconstructed runtime configuration.

After prerequisites pass, select one scanned component of a promoted release and record its immutable image digest, exact package PURL, vulnerability, product key, tenant and environment. Preserve every PURL qualifier: a repository-qualified product and its unqualified counterpart are distinct overlay identities. Record and edit the component decision through Org decisions. Capture the rendered console’s decision request and the subsequent Policy effective status (overlayApplied: true), Findings displayed vexStatus, and actual promotion/gate outcome. Changing that decision to UNDER_INVESTIGATION withdraws its status assertion; prove that the same reads and gate return to the baseline. Record a separate case-trust hold/withdrawal control and the tenant/environment isolation of a remembered issuer preference. A successful API call or prerequisite result is not this behavioral proof.

Capture the changed Findings and gateway OpenAPI documents from the upgraded running hosts through tools/scripts/docs/regen-from-openapi.ps1, with their changelog and truthfulness check. Do not rewrite an older capture to imply that a source-only upgrade is deployed.

8. Related documents

• VEX Consensus Troubleshooting Guide
• Vulnerabilities hub architecture: statements, distro release scoping, API surface and events
• Policy architecture: gate projection, effective VEX status read, SmartDiff:Gates
• Findings architecture: effective VEX status on findings
• stella vex command guide