Trust Lattice Operations Runbook

Version: 1.2.0 Last Updated: 2026-05-31 Audience: Operations and Support teams

This runbook is the operator-facing guide for running the Stella Ops Trust Lattice — the VEX claim-scoring framework that turns advisory and VEX evidence into explainable, deterministic verdicts. It covers monitoring, configuration, common operations, and emergency procedures. For symptom-driven diagnosis see the companion Trust Lattice Troubleshooting Guide; for the design contract see the module dossier docs/modules/excititor/trust-lattice.md.

Reconciliation note (2026-05-30, re-verified 2026-05-31): This runbook was reconciled against src/ ground truth. Several command verbs, metric names, and environment variables described in the 1.0 draft were aspirational and are not implemented in the current code. They are flagged inline as NOT IMPLEMENTED or replaced with the real surface so operators do not paste commands that fail. Trust-vector scoring, claim merging, the policy gates, the signed verdict manifest, and trust calibration all exist in source and are documented accurately below.

The 2026-05-31 deep pass corrected several errors the 1.1 reconciliation introduced or missed: (1) the .sample config templates live under devops/etc/, not a (non-existent) root etc/; (2) a top-level stella verdict command does exist (verify/list/push/rationale), and a stella gate group exists for CI/CD gate evaluation — the 1.1 note that implied neither existed was wrong; (3) the calibration audit tables use schema excititorwith table calibration_adjustments(per 001_initial_schema.sql / ExcititorDbContext), not the excititor_calibration schema / trust_vector_adjustments table named in the YAML sample’s storage: block; (4) source paths now include the real __Libraries/ segment. The authoritative module dossier is docs/modules/excititor/trust-lattice.md.

1. Overview

The Trust Lattice is a VEX claim scoring framework that produces explainable, deterministic verdicts. This runbook covers operational procedures for monitoring, troubleshooting, and maintaining the system.

The scoring core (trust vectors, scorers, claim-score merge, and calibration) lives in StellaOps.Excititor.Core, which after the module consolidation is housed under src/Concelier/__Libraries/StellaOps.Excititor.Core. Runtime VEX normalization, signature verification, and consensus projection are performed by VexLens (src/VexLens). The policy gates that enforce trust thresholds live in StellaOps.Policy (src/Policy/__Libraries/StellaOps.Policy), and signed verdict manifests are produced and replay-verified by Authority (src/Authority/__Libraries/StellaOps.Authority.Core/Verdicts).

2. System Components

Source of truth verified against src/ on 2026-05-30. “Excititor” is no longer a standalone service directory; its core libraries are consolidated under src/Concelier/__Libraries/StellaOps.Excititor.*.

Scopes (from StellaOpsScopes.cs): read paths use vex:read; VEX provider ingestion/review uses vex.admin; raw VEX ingest uses vex:ingest. Policy-gate authoring and runs use the policy:* family (policy:read, policy:author, policy:run, etc.). There is no dedicated trustlattice:* or calibration:* scope — calibration and gate tuning are gated through vex.admin / policy:*.

3. Monitoring

3.1 Key Metrics

Reconciled 2026-05-30. The trustlattice_*, policy_gate_failures_total, verdict_manifest_replay_failures, and calibration_drift_percent metric names from the 1.0 draft are NOT IMPLEMENTED — no src/ code emits them. The metrics actually exported today are listed below. Where a draft metric has no real counterpart it is flagged as a roadmap item rather than removed, so the dashboard backlog stays visible.

Emitted today (verified in source):

Draft metrics NOT IMPLEMENTED (roadmap — do not alert on these yet):

3.2 Dashboards

The Grafana dashboard UID trustlattice and the PromQL queries below reference the NOT IMPLEMENTED draft metrics from §3.1; they will not return data against current builds. Retained as the dashboard backlog. Build live dashboards from the emitted vexlens.* / excititor_vex_* / analysis.vexgate.* families.

• Grafana: https://<grafana>/d/trustlattice (dashboard not yet provisioned)
• Prometheus queries (draft — metrics not yet emitted):

  # Average claim score by source class — NOT IMPLEMENTED (no trustlattice_claim_score series)
  avg(trustlattice_claim_score) by (source_class)
  
  # Gate failure rate — NOT IMPLEMENTED (no policy_gate_failures_total series)
  rate(policy_gate_failures_total[5m])
  
  # Confidence distribution — use the live VexLens series instead:
  histogram_quantile(0.5, sum(rate(vexlens_consensus_confidence_bucket[5m])) by (le))
  

3.3 Log Queries

Key log entries (Loki/ELK). The excititor app label corresponds to the StellaOps.Excititor.WebService deployment; consensus events are also emitted by the vexlens deployment:

# Consensus / claim scoring (VexLens log events: ConsensusCompleted, ConflictDetected)
{app="vexlens"} |= "Consensus"

# VEX signature verification
{app="excititor"} |= "verification"

# Gate evaluation (Policy)
{app="policy"} |= "gate"

# Verdict replay verification (Authority — VerdictReplayVerifier)
{app="authority"} |= "replay"

4. Common Operations

Reconciled 2026-05-30. The stella trustvector, stella verdict, stella gates, and stella calibration command groups from the 1.0 draft do NOT exist in the CLI (src/Cli/StellaOps.Cli). The real, verified verbs are shown below. Trust-vector defaults are configured via file (see §6), not via a CLI mutation command. Calibration runs as a scheduled service epoch, not an ad-hoc CLI invocation.

4.1 Viewing VEX Consensus and Trust Decisions

# List VEX consensus decisions (real command — stella vex consensus list)
stella vex consensus list --vuln-id CVE-2025-12345 --status not_affected

# Show detailed consensus incl. quorum, evidence, rationale, signature status
stella vex consensus show CVE-2025-12345 <product-key>

The draft stella trustvector list --source-class vendor is NOT IMPLEMENTED. Default trust vectors per source classification are defined in devops/etc/trust-lattice.yaml.sample (§6.1) and in code defaults (DefaultTrustVectors.GetDefault(provider.Kind), consumed by TrustCalibrationService). There is no REST endpoint at /api/v1/trustlattice/vectors.

4.2 Inspecting a Verdict

# Top-level verdict command group (verified in VerdictCommandGroup.cs):
stella verdict verify <reference> [--decision pass|warn|block] [--strict] [--verify-uncertainty]
stella verdict list <reference>          # list verdict attestations for an image
stella verdict push <reference> -f <dsse-envelope.json>
stella verdict rationale <finding-id>    # 4-line rationale (Evidence/Policy/Attestations/Decision)

# Manage VEX decisions (DSSE-signed, Rekor transparency) — stella decision
stella decision --help

# Deterministic verdict replay (Authority VerdictReplayVerifier surface):
stella replay --help          # "Replay scans from run manifests and compare verdicts"
stella evidence replay --help # "Deterministic verdict replay."
stella score replay --help    # "Replay a score computation for a scan"

Corrected 2026-05-31. A top-level stella verdict command group does exist (VerdictCommandGroup.cs) with the subcommands verify, list, push, and rationale shown above — the earlier 1.1 note that claimed there was no top-level verdict command was wrong. What is genuinely NOT IMPLEMENTED are the specific draft verbs stella verdict show <id> and stella verdict replay <id> (no show subcommand; replay lives under the replay / evidence replay / score replay groups, not under verdict). Verdict manifests are built and replay-verified inside Authority (src/Authority/__Libraries/StellaOps.Authority.Core/Verdicts, VerdictReplayVerifier); the decision group manages signed VEX decisions. CLI telemetry uses activity names cli.verdict.list, cli.verdict.verify, cli.verdict.push, and cli.verdict.rationale.

4.3 Viewing Gate Configuration

# Gate *configuration* is file-driven, not a CLI query.
# Inspect the active policy-gates config (deployed copy of the sample):
cat policy-gates.yaml   # template: devops/etc/policy-gates.yaml.sample

# A `stella gate` command group DOES exist, but for CI/CD gate *evaluation*,
# not for printing gate config (GateCommandGroup.cs):
stella gate evaluate --help   # run release gates in a CI pipeline
stella gate status --help     # query gate status
stella gate score --help      # score-based gate evaluation

Corrected 2026-05-31. The draft stella gates list / stella gates show verbs (plural gates, for reading config) are NOT IMPLEMENTED. There is, however, a real stella gate (singular) command group for CI/CD gate evaluation with evaluate, status, and score subcommands (GateCommandGroup.cs) — it does not list or print the gate-config file. Gate definitions, thresholds, and per-environment values live in the deployed policy-gates.yaml (template: devops/etc/policy-gates.yaml.sample). See §6.1 for the full gate set.

4.4 Triggering Manual Calibration

The draft stella calibration run/history verbs are NOT IMPLEMENTED. Trust calibration is performed by TrustCalibrationService on a scheduled epoch (default intervalDays: 30, minimumSamples: 100 — see devops/etc/excititor-calibration.yaml.sample). Calibration manifests and adjustments are persisted to the excititor schema for audit: excititor.calibration_manifests, excititor.calibration_adjustments, and excititor.source_trust_vectors (verified in StellaOps.Excititor.Persistence/Migrations/001_initial_schema.sql and the ExcititorDbContext table mappings). Caveat: the shipped .sample storage: block names a excititor_calibration schema and a trust_vector_adjustments table — those names do not match the actual migration/EF mappings (schema excititor, table calibration_adjustments); trust the migration, not the sample. There is no ad-hoc CLI trigger; tuning is done by editing the calibration config and restarting the worker, or by toggling calibration.enabled.

5. Emergency Procedures

Reconciled 2026-05-30. Steps below were rewritten to use real commands and live signals. The stella vex source status, stella verdict list/diff, and stella calibration freeze/history/rollback/unfreeze verbs are NOT IMPLEMENTED and have been replaced with file/DB/service procedures that work against current builds.

5.1 High Gate Failure Rate

Symptoms:

• Rising gate-rejection volume in analysis.vexgate.results (the named policy_gate_failures_total counter is NOT IMPLEMENTED — use vexgate results and gate evaluation logs)
• Many builds failing due to low confidence

Steps:

1. Check whether a VEX source is stale or unavailable. Inspect VEX consensus for the affected source rather than a nonexistent vex source status verb:

   stella vex consensus list --status under_investigation --limit 20
   

2. If a source is stale, consider a temporary threshold reduction in the deployed gate config (template: devops/etc/policy-gates.yaml.sample):

   # deployed policy-gates.yaml
   gates:
     minimumConfidence:
       thresholds:
         production: 0.60  # Reduced from 0.75 (sample default)
   

3. Restart the Policy Engine to apply changes

4. Monitor and restore the threshold once the source recovers

5.2 Verdict Replay Failures

Symptoms:

• Verdict replay mismatches reported by Authority’s VerdictReplayVerifier (the verdict_manifest_replay_failures counter is NOT IMPLEMENTED — detect via Authority replay logs / audit checks)
• Audit compliance check failures

Steps:

1. Re-run a deterministic verdict replay through the supported CLI surface:

   stella replay --help          # replay scans from run manifests, compare verdicts
   stella evidence replay --help # deterministic verdict replay
   

2. Compare original vs replayed inputs. VerdictReplayVerifier compares the recorded VerdictManifest inputs against a fresh evaluation; surface the diff through the replay command output (there is no stella verdict diff).

3. Common causes:

   • VEX document modified after the verdict was sealed
   • Clock drift during evaluation
   • Policy configuration changed between seal and replay

4. For clock drift, verify NTP synchronization:

   timedatectl status
   

5.3 Trust Vector Drift Emergency

Symptoms:

• Large/sudden trust-vector adjustments (the calibration_drift_percent gauge is NOT IMPLEMENTED — inspect the excititor.calibration_adjustments table and calibration manifests instead)
• Sudden confidence changes across many assets

Steps:

1. Freeze calibration by disabling it in the deployed calibration config and restarting the worker (there is no stella calibration freeze verb):

   # deployed excititor-calibration.yaml (template: devops/etc/excititor-calibration.yaml.sample)
   calibration:
     enabled: false
   

2. Investigate recent calibration epochs from the audit tables. The actual schema is excititor (NOT excititor_calibration) and the adjustments table is calibration_adjustments (NOT trust_vector_adjustments); it has no timestamp column, so order adjustments via the manifest epoch:

   -- excititor schema (per 001_initial_schema.sql)
   SELECT * FROM excititor.calibration_manifests ORDER BY epoch_number DESC LIMIT 5;
   SELECT a.*
     FROM excititor.calibration_adjustments a
     JOIN excititor.calibration_manifests m ON m.manifest_id = a.manifest_id
    ORDER BY m.epoch_number DESC
    LIMIT 50;
   

3. If false-positive rate increased, rely on the built-in rollback: the calibration config exposes rollback.enabled with a degradationThreshold (default 10%) and evaluationWindowDays (default 7). There is no manual stella calibration rollback verb — recover a prior epoch via PostgreSQL snapshot restore if automatic rollback did not trigger (see docs/runbooks/migration-recovery.md).

4. Re-enable calibration after investigation by setting calibration.enabled: true and restarting the worker.

6. Configuration

6.1 Configuration Files

The repo ships .sampletemplates under devops/etc/(verified 2026-05-31 — there is no top-level etc/ directory in the repo root). Operators copy each to the non-.sample path the service reads. Template directory corrected 2026-05-31.

Default trust weights (devops/etc/trust-lattice.yaml.sample, matching TrustWeights code defaults):

Formula: BaseTrust = (WP × P) + (WC × C) + (WR × R). Freshness decay default half-life is 90 days with a floor of 0.35 (FreshnessCalculator); the devops/etc/policy-gates.yaml.sample exposes freshness.floor: 0.35, while devops/etc/trust-lattice.yaml.sample documents minimumFreshness: 0.5. Conflict penalty default is 0.25 — MergePolicy.ConflictPenalty on the Policy side (src/Policy/__Libraries/StellaOps.Policy/TrustLattice/ClaimScoreMerger.cs) and the conflictPenalty constructor parameter (also defaulting to 0.25) on the Excititor ClaimScoreMerger.

Gate set (devops/etc/policy-gates.yaml.sample; classes under src/Policy/__Libraries/StellaOps.Policy/Gates):

6.2 Environment Variables

Reconciled 2026-05-30. The 1.0 draft variable names (TRUSTLATTICE_WEIGHTS_*, GATES_MINIMUM_CONFIDENCE_PROD, CALIBRATION_LEARNING_RATE) are NOT the names used. The only env-var override names documented in the shipped samples are the STELLAOPS_* names below. Caveat: these appear as commented suggestions in the .sample files; a code search did not find a binding that reads them, so treat them as the intended override surface and verify against your build before relying on them. File-based config (§6.1) is the verified configuration path.

Learning-rate caveat: the calibration sample sets learningRate.base: 0.15 and maxAdjustmentPerEpoch: 0.25, whereas the TrustVectorCalibrator code default is LearningRate = 0.02 / MaxAdjustmentPerEpoch = 0.05. The 1.0 draft’s 0.02 matches the code default, not the shipped sample. Confirm which layer wins in your deployment before tuning.

Production minimumConfidence (0.75) is set in the deployed policy-gates.yaml (template devops/etc/policy-gates.yaml.sample) under gates.minimumConfidence.thresholds.production, not via a GATES_MINIMUM_CONFIDENCE_PROD env var (NOT IMPLEMENTED).

7. Maintenance Tasks

7.1 Daily

• [ ] Review gate failure alerts
• [ ] Check verdict replay success rate
• [ ] Monitor trust vector stability

7.2 Weekly

• [ ] Review calibration epoch results
• [ ] Analyze conflict rate trends
• [ ] Update trust vectors for new sources

7.3 Monthly

• [ ] Audit high-drift sources
• [ ] Review and tune gate thresholds
• [ ] Clean up expired verdict manifests

8. Contact & Related Documentation

• On-call: #trustlattice-oncall (Slack)
• Escalation: VEX Guild Lead
• Module dossier (design contract): docs/modules/excititor/trust-lattice.md
• Troubleshooting (symptom-driven): Trust Lattice Troubleshooting Guide
• VEX consensus model: VEX Consensus Guide

Document Version: 1.2.0 (reconciled against src/ on 2026-05-30; deep re-verified 2026-05-31) Sprint: 7100.0003.0002