Trust Lattice Troubleshooting Guide

Version: 1.2.0 Last Updated: 2026-05-31 Audience: Support and Development teams

This is the symptom-driven companion to the Trust Lattice Operations Runbook. Use the Quick Reference table to jump from a symptom (low confidence, gate failures, replay mismatches, calibration drift, claim conflicts) to the section that explains the cause and the diagnostic commands. The authoritative design contract is the module dossier docs/modules/excititor/trust-lattice.md.

Reconciliation note (2026-05-31, deep pass): This guide was reconciled again, command-by-command, against the implemented code. The trust-lattice scoring model it describes is real — the Provenance/Coverage/Replayability (PCR) trust vector, freshness decay, claim-strength multipliers, conflict penalty, and the four policy gates all exist (see Implementation map) and every numeric default below was verified against source. The v1.2.0 pass corrected several lingering drifts: config samples live under devops/etc/(there is no top-level etc/), the VexLens HTTP API has no export endpoint, stella vex explain takes --product-key as a required option (not a positional) and has no --full-evidence flag, the gate-scan CLI command is gate-scan(hyphenated), and the ReachabilityRequirementGate also fires on HIGH severity by default. Commands and endpoints are annotated:

• (implemented) — the command/endpoint exists today.
• (illustrative — not implemented) — kept as a worked example of the diagnostic intent; there is no such command/endpoint. Where a real equivalent exists, it is named inline.

The authoritative spec is docs/modules/excititor/trust-lattice.md; the companion operational runbook is docs/operations/trust-lattice-runbook.md.

Quick Reference

1. Diagnostic Commands

1.1 Check System Health

VEX consensus scoring runs in VexLens (src/VexLens), with the PCR trust vector and policy gates living in Concelier/Excititor.Core and the Policy Engine. Each service exposes a standard /health check.

# VexLens health (consensus + trust scoring)  (implemented — /health)
curl https://api.example.com/vexlens/health

# Policy Engine health  (implemented — /health)
curl https://api.example.com/policy/health

# Concelier/Excititor health  (implemented — /health)
curl https://api.example.com/concelier/health

# Authority health  (implemented — /health)
curl https://api.example.com/authority/health

The health route is /health on each service (MapHealthChecks("/health", …)), returning { "status": …, "checks": [...] }. Path prefixes shown above are the gateway route prefixes; confirm against your gateway route table.

1.2 Trace a Verdict / Consensus Decision

# Show a VEX consensus decision (quorum, evidence, rationale, signature status)
#   (implemented) — vuln-id and product-key are positional args here
stella vex consensus show <vulnerability-id> <product-key> --full-evidence

# Explain a VEX decision with reachability evidence + verification status
#   (implemented) — product-key is a REQUIRED option, not positional
stella vex explain <vulnerability-id> --product-key <product-key>

# Preview a consensus with trust/threshold overrides (implemented)
stella vex simulate --vuln-id <vulnerability-id> --product-key <product-key> \
  --threshold 0.7 --trust nvd=1.5 --exclude somehub

# A consensus/explain decision surfaces:
# - The VEX statements (claims) considered
# - Trust / quorum weighting per source
# - Conflict detection and rationale
# - Signature / verification status and DSSE pointers

(illustrative — not implemented) There is no stella verdict explain <manifestId> command. The stella verdict group is for verdict attestations (verify, list, push, rationale) on container images, not for trust-lattice trace inspection. Use stella vex consensus show / stella vex explain (above) for claim-level explanation.

1.3 Check VEX Source / Provider Status

# List Excititor VEX providers (definitions, overrides, runtime readiness)
#   (implemented)
stella vex providers list

# Show readiness, trust metadata, and persisted state for one provider
#   (implemented)
stella vex providers show <provider-id>

# Report readiness/health for VEX providers without triggering a run
#   (implemented)
stella vex providers check

(illustrative — not implemented) The earlier stella vex source list / stella vex source status vendor:redhat commands do not exist. The real source-management surface is stella vex providers … (above; the full set is configure, artifacts, list, show, enable, disable, sync, check). Real provider IDs are excititor:-prefixed — e.g. excititor:redhat, excititor:cisco, excititor:msrc, excititor:oci-openvex (short aliases redhat/cisco/msrc/oci are accepted). Provider class (vendor / distro / hub / platform / internal / attestation) comes from SourceClassificationService; the vendor:/hub: style identifiers used elsewhere in this guide are illustrative class labels, not literal provider IDs.

2. Common Issues

2.1 Low Confidence Scores

Symptoms:

• Verdicts have confidence < 0.5
• Many “under_investigation” statuses

Diagnosis:

The commands in steps 1 and 3 below are illustrative — not implemented under those names. Use stella vex consensus show <vuln-id> <product-key> --full-evidence (implemented) to inspect the claims, per-source weighting, and freshness behind a low-confidence decision; use stella vex providers show <provider-id> (implemented) to inspect a source’s trust metadata.

1. Check claim freshness (what to look for in the consensus evidence):

   # Freshness uses an exponential decay: F = max(2^(-ageDays/90), 0.35).
   # The half-life is 90 days and the floor is 0.35 (FreshnessCalculator).
   # A claim hits the 0.35 floor only when it is many half-lives old; freshness
   # never drops below 0.35 unless the claim is revoked.
   # Look for: stale claims pinned at the floor, or no high-trust sources.
   

2. Check trust vector values:

   # (implemented) inspect a provider's trust metadata and persisted state
   stella vex providers show <provider-id>
   
   # Low PCR component scores indicate:
   # - Signature / transparency-log issues lower Provenance (P)
   # - Coarse scope (product/family-level only) lowers Coverage (C)
   # - Unpinned / ephemeral inputs lower Replayability (R)
   # Base trust = 0.45*P + 0.35*C + 0.20*R (default weights).
   

3. Check for missing VEX coverage:

   # (illustrative — not implemented) there is no `stella vex coverage` command.
   # Instead, list consensus decisions and filter by PURL:
   stella vex consensus list --purl pkg:npm/lodash@4.17.21
   # No decision returned -> no source covers this package.
   

Resolution:

• If freshness is low: Check if the source is publishing updates
• If a trust vector component is low: Review the provider’s verification settings (signing, transparency log) and scope-matching
• If coverage is missing: Add or enable additional VEX providers

2.2 Gate Failures

Symptoms:

• Builds failing with “Gate: MinimumConfidenceGate FAILED”
• Policy violations despite VEX claims

The four gates below are real and ship with the Policy Engine (src/Policy/__Libraries/StellaOps.Policy/Gates/): MinimumConfidenceGate, SourceQuotaGate, UnknownsBudgetGate, ReachabilityRequirementGate. Their defaults are baked into the gate option records and can be overridden via devops/etc/policy-gates.yaml.sample. (The Policy Engine ships several more gates in that directory — e.g. CvssThresholdGate, EvidenceFreshnessGate, FacetQuotaGate, SignatureRequiredGate, SbomPresenceGate, VexProofGate — but only the four trust-lattice gates above are in scope for this guide. The sample file also defines a vexTrust gate keyed on the VexLens composite score.)

Diagnosis:

(illustrative — not implemented) There is no stella gates show, stella verdict show, or stella verdict gates command. Gate thresholds are read from devops/etc/policy-gates.yaml.sample (and per-policy overrides); gate outcomes are surfaced through the consensus/gating APIs and the stella vex gate-scan / stella vex consensus show output (note: the CLI subcommand is gate-scan, hyphenated — stella vex gatescan does not parse). The blocks below document the real default values and pass/fail logic.

1. Default thresholds (MinimumConfidenceGate, applies to not_affected and fixed statuses; affected always bypasses):

   production:  0.75
   staging:     0.60
   development: 0.40
   

2. Compare with the decision confidence:

   # confidence: 0.68  <- below the 0.75 production threshold => gate FAILS
   

3. Default gate set and example outcome:

   #   MinimumConfidenceGate: FAILED (0.68 < 0.75)
   #   SourceQuotaGate: PASSED      (top source <= 60% influence, or corroborated)
   #   UnknownsBudgetGate: PASSED   (#unknowns <= 5; cumulative uncertainty <= 2.0)
   #   ReachabilityRequirementGate: PASSED
   #     (applies only to not_affected at severity >= the threshold; default
   #      SeverityThreshold=CRITICAL. With RequireSubgraphProofForHighSeverity=true
   #      (the default) it also kicks in for HIGH (severityRank >= 3) and then
   #      *requires a subgraph proof* — it FAILS not_affected on HIGH/CRITICAL when
   #      no SubgraphSlice is supplied, unless a bypass reason such as
   #      component_not_present / vulnerable_configuration_unused is present.)
   

Resolution:

• Temporary: Lower the threshold in devops/etc/policy-gates.yaml.sample (with approval)
• Long-term: Add corroborating VEX sources
• If single-source: Check SourceQuotaGate — a top source above 60% influence passes only if a second source is within corroborationDelta (0.10)

2.3 Verdict Replay Failures

Symptoms:

• Replay verification returns success: false
• Audit failures due to non-determinism

(illustrative — not implemented) There is no stella verdict replay command. Deterministic VEX replay is exposed as an HTTP endpoint on Concelier: GET /concelier/advisories/{vulnerabilityKey}/replay (implemented). The CLI stella scan replay command (implemented) replays a scan with explicit hashes for deterministic verdict reproduction. The diff/cause tables below document the replay semantics regardless of the entry point.

Diagnosis:

1. Get a detailed diff (semantics):

   # A replay re-derives the decision from pinned inputs and reports differences:
   #   result.confidence: 0.82 -> 0.79
   #   inputs.vexDocumentDigests[2]: sha256:abc... (missing)
   

2. Common causes:

   Difference	Likely Cause
   VEX digest mismatch	Document was modified after verdict
   Confidence delta	Clock cutoff drift (freshness calc)
   Missing claims	Source was unavailable during replay
   Different status	Policy version changed

3. Check input availability:

   # (illustrative — not implemented) there is no `stella cas verify` command.
   # Confirm the pinned inputs (SBOM digests, VEX document digests, feed
   # snapshot IDs, reachability graph IDs, policy hash, lattice version, clock
   # cutoff) are still resolvable in CAS / storage before treating a mismatch
   # as drift.
   

Resolution:

• Clock drift: Ensure NTP synchronization across nodes
• Missing inputs: Restore from backup or acknowledge drift
• Policy change: Compare policy hashes between original and replay

2.4 Calibration Issues

Symptoms:

• Trust vectors changed unexpectedly
• Accuracy metrics declining

Calibration is implemented in CalibrationComparisonEngine / TrustVectorCalibrator (src/Concelier/__Libraries/StellaOps.Excititor.Core/Calibration/). It compares per-source observations against a truth set over an epoch (time window) and reports Accuracy, CorrectPredictions, FalsePositives, FalseNegatives, a 95% ConfidenceInterval, and a DetectedBias (None, OptimisticBias, PessimisticBias, ScopeBias).

Diagnosis:

(illustrative — not implemented) There is no stella calibration … command group. Calibration runs as a backend service over an epoch window; the comparison results below reflect the real ComparisonResult shape and bias-detection logic, but the stella calibration history|epoch|validate-truth invocations are illustrative.

1. Review recent calibration results (per-source comparison output):

   # Accuracy = CorrectPredictions / TotalPredictions
   # Epoch (window): accuracy=0.92, CI(95%)=±0.014
   

2. Check comparison results (ComparisonResult fields):

   # TotalPredictions: 1500
   # CorrectPredictions: 1380
   # FalsePositives: 45
   # FalseNegatives: 75
   # DetectedBias: OptimisticBias
   #   DetectBias() evaluates IN THIS ORDER and returns the first match:
   #   1) ScopeBias        : scopeMismatches >= max(2, total/3)   (checked first)
   #   2) OptimisticBias   : FalseNegatives >= 2 and FN > FalsePositives
   #   3) PessimisticBias  : FalsePositives >= 2 and FP > FalseNegatives
   #   4) None
   #   Because ScopeBias is tested first, a source with many scope mismatches is
   #   labelled ScopeBias even if it would otherwise look Optimistic/Pessimistic.
   #   (FalseNegative = predicted not-affected but truth = affected;
   #    FalsePositive = predicted affected but truth = not-affected.)
   

3. Check for data quality issues:

   # Observations without a matching truth entry are skipped, not scored.
   # Inspect the truth set for the epoch for corrupted/missing entries before
   # acting on a bias finding.
   

Resolution:

• High false positive (PessimisticBias): Review the source’s provenance/strength weighting
• High false negative (OptimisticBias): Review coverage / scope matching
• ScopeBias: Tighten product-mapping (CPE/PURL) for the source
• Data quality issue: Re-run the epoch with a corrected truth set

2.5 Claim Conflicts

Symptoms:

• Verdicts show hasConflicts: true
• Confidence reduced due to conflict penalty

Conflict handling is real (ConflictPenalizer, src/Policy/__Libraries/StellaOps.Policy/TrustLattice/). When claims for the same (CVE, Asset) disagree on status, the strongest claim (by original score, then scope specificity, then source ID) keeps its score; every claim with a different status is multiplied by (1 - ConflictPenalty) where ConflictPenalty = 0.25.

Diagnosis:

(illustrative — not implemented) There is no stella verdict conflicts, stella vex claim, or stella claim compare command. Conflicts are surfaced through the consensus APIs and the --json output of the implemented commands below. The penalty value and tie-break order shown are the real defaults.

1. View conflict details:

   # (implemented) decision-level conflict + rationale
   stella vex consensus show <vulnerability-id> <product-key> --json
   # Or list decisions flagged with conflicts:
   stella vex consensus list --status under_investigation
   #
   # Conflict penalty applied to dissenting claims: 0.25
   

2. Investigate source disagreement:

   # (implemented) explain the decision with per-source/reachability evidence.
   # NOTE: product-key is a REQUIRED option (--product-key / -p), not a
   # positional argument; explain has NO --full-evidence flag (that flag lives on
   # `consensus show`). explain instead defaults --call-paths/--runtime-hits/
   # --graph to true and adds --dsse/--rekor/--verify/--offline toggles.
   stella vex explain <vulnerability-id> --product-key <product-key>
   

3. Check claim timestamps:

   # Freshness decay (90-day half-life) already down-weights older claims in
   # the score; compare the per-source `issuedAt` values in the explain output.
   

Resolution:

• If one source is stale: Flag for review (freshness already down-weights it)
• If genuine disagreement: Higher-trust source wins (by design)
• If persistent: Consider a source/provider override in policy

3. Performance Issues

3.1 Slow Claim Scoring

Symptoms:

• Scoring latency > 100ms
• Timeouts during high load

Diagnosis:

# (illustrative — not implemented) there is no `stella perf scoring` command.
# Use service metrics/telemetry (VexLensMetrics, Observability) and the
# consensus-rationale cache to investigate scoring latency. Look for:
# - Consensus-rationale cache miss rate
# - Trust vector / provider lookups
# - Freshness calculation overhead

Resolution:

• Warm the consensus-rationale cache (IConsensusRationaleCache)
• Pre-compute freshness for common clock cutoffs
• Scale VexLens horizontally

3.2 Slow Verdict Replay

Symptoms:

• Replay verification > 5 seconds
• Timeout during audit

Diagnosis:

# (illustrative — not implemented) there is no `stella verdict replay --timing`
# command. Replay timing is dominated by input retrieval; an illustrative split:
#   Input fetch: 3.2s   <- usually the bottleneck (CAS / remote storage)
#   Score compute: 0.1s
#   Merge: 0.05s
#   Total: 3.35s

Resolution:

• Ensure CAS storage is local or cached
• Pre-warm the consensus/projection cache for critical assets
• Increase timeout for large manifests

4. Integration Issues

4.1 VEX Source Not Recognized

Symptoms:

• Claims from source not included in verdicts
• Source shows as “unknown” class

Resolution:

1. Register the source in configuration. The shipped sample (devops/etc/trust-lattice.yaml.sample) is keyed by provider class, not by an explicit per-source list; per-class default vectors live under defaultVectors::

   # devops/etc/trust-lattice.yaml   (sample: devops/etc/trust-lattice.yaml.sample)
   defaultWeights:
     provenance: 0.45
     coverage: 0.35
     replayability: 0.20
   defaultVectors:
     vendor:
       provenance: 0.90
       coverage: 0.85
       replayability: 0.70
   

   Code is the source of truth for class defaults. DefaultTrustVectors.cs (PCR): vendor 0.90/0.70/0.60, distro 0.80/0.85/0.60, internal/Platform 0.85/0.95/0.90, hub 0.60/0.50/0.40, attestation 0.95/0.80/0.70. NOTE: the defaultVectors: numbers in the sample YAML differ from these compiled defaults (e.g. sample vendor 0.90/0.85/0.70 vs code 0.90/0.70/0.60) and the sample has no attestation class — the compiled DefaultTrustVectors wins unless a runtime config actually overrides it. Reconcile the sample against the code before relying on its values.

2. Apply configuration:

   # (illustrative — not implemented) there is no `stella config reload` command.
   # Trust-lattice / provider configuration is reloaded by restarting the owning
   # service, or by updating persisted provider config via:
   #   stella vex providers configure <provider> --set key=value   (implemented)
   #   (provider accepts excititor:redhat etc., or short aliases redhat/cisco/...)
   

4.2 Gate Not Evaluating

Symptoms:

• Expected gate not appearing in results
• Gate shows as “disabled”

Resolution:

1. Check gate configuration:

   # (illustrative — not implemented) there is no `stella gates list` command.
   # Gate enable/disable flags live in the gate config file; each gate's
   # `Enabled` option defaults to true in code.
   

2. Enable gate:

   # devops/etc/policy-gates.yaml  (sample: devops/etc/policy-gates.yaml.sample)
   gates:
     minimumConfidence:
       enabled: true  # Ensure this is true
   

5. Support Information

5.1 Collecting Diagnostic Bundle

# (illustrative — not implemented) there is no `stella support bundle` command
# for trust-lattice. Collect diagnostics manually:
#   - export the relevant consensus decisions via `stella vex export` (CLI,
#     implemented). NOTE: there is NO `/api/v1/vexlens/export` HTTP endpoint —
#     the VexLens service exposes consensus/projection/conflict/stats reads:
#       POST /api/v1/vexlens/consensus            (compute consensus)
#       POST /api/v1/vexlens/consensus:withProof  (consensus + full proof)
#       GET  /api/v1/vexlens/projections          (query stored projections)
#       GET  /api/v1/vexlens/projections/history  (projection history for a pair)
#       GET  /api/v1/vexlens/conflicts            (projections with conflicts)
#       GET  /api/v1/vexlens/stats                (aggregate statistics)
#     (all under scope vexlens.read; issuer admin endpoints need vexlens.write)
#   - capture provider state with `stella vex providers list --json`
#   - capture service logs and metrics (see 5.2)

A diagnostic collection should include:

• Provider / trust metadata snapshots (stella vex providers list)
• Recent consensus decisions (stella vex consensus list, or stella vex export)
• Gate evaluation results
• Calibration comparison results
• System metrics

5.2 Log Locations

Log paths are deployment-dependent (container stdout/Serilog by default). The table below lists the services that own the trust-lattice pipeline; adjust the paths to your deployment. Note the module consolidation: the Excititor trust-vector/calibration code ships inside Concelier (src/Concelier/__Libraries/StellaOps.Excititor.Core), and consensus scoring runs in VexLens (src/VexLens).

5.3 Contact

• Support: support@stella-ops.org
• Spec / dossier: docs/modules/excititor/trust-lattice.md(note: its §10 source paths still reference src/Excititor/…; the code now lives under src/Concelier/__Libraries/StellaOps.Excititor.Core/ and src/Policy/__Libraries/StellaOps.Policy/)
• Operational runbook: docs/operations/trust-lattice-runbook.md
• GitHub Issues: https://github.com/stella-ops/stella-ops/issues

6. Implementation Map

The model this guide troubleshoots is implemented (module consolidation means Excititor code lives inside Concelier):

Two trust models coexist. Sections 1–5 describe the PCR trust vector (Provenance / Coverage / Replayability) used by the lattice merge and gates. VexLens additionally maintains a 5-component source-trust score — Authority (0.25), Accuracy (0.30), Timeliness (0.15), Coverage (0.10), Verification (0.20) — for consensus weighting (VexSourceTrustScore). Both use a 90-day half-life / 0.35 floor decay. Do not conflate the two when reading scores.

Document Version: 1.2.0 Reconciled: 2026-05-31 (doc↔code, deep command-by-command pass) Original Sprint: 7100.0003.0002