Proof Verification Operations Runbook
Version: 1.2.0
Sprint: 3500.0004.0004
Last Updated: 2026-05-31
Audience: operators and auditors verifying Stella Ops evidence — attestation bundles and proof spines — on the CLI or via the Scanner API.
This runbook covers operational procedures for proof verification across the two proof surfaces that exist in the codebase today:
- Attestation bundle verification via the
stella proof verifyCLI (StellaOps.Cli,ProofCommandGroup→AttestationBundleVerifier). This verifies a.tar.gzbundle exported from the Export Center, containing a DSSE envelope wrapping an in-toto statement. - Proof Spine inspection via the Scanner WebService HTTP API (
ProofSpineEndpoints→ProofSpineVerifier). A proof spine is an ordered, hash-chained sequence of typed evidence segments (SBOM slice → match → reachability → policy eval …) for a single artifact/vulnerability/profile tuple.
Source-of-truth note. Several procedures in earlier drafts described a 4-leaf Merkle tree of
sbom/rules/policy/feedhashes, X.509 certificate-chain validation, and aPOST .../proofs/{rootHash}/verifyendpoint. None of those exist insrc/. This revision reconciles the runbook against the actual code:src/Cli/StellaOps.Cli/Commands/Proof/,src/Cli/StellaOps.Cli/Services/AttestationBundleVerifier.cs,src/Scanner/StellaOps.Scanner.WebService/Endpoints/ProofSpineEndpoints.cs,src/Scanner/__Libraries/StellaOps.Scanner.ProofSpine/, andsrc/Scanner/__Libraries/StellaOps.Scanner.Core/ProofBundleWriter.cs.
Table of Contents
- Overview
- Attestation Bundle Verification (CLI)
- Proof Spine Verification (API)
- Offline Verification
- Transparency Log Integration
- Troubleshooting
- Monitoring & Alerting
- Escalation Procedures
1. Overview
What is Proof Verification?
Proof verification cryptographically validates that StellaOps evidence has not been tampered with. Two independent surfaces implement it:
Attestation bundle (CLI) — AttestationBundleVerifier checks, in order:
Bundle existence and (when a co-located
<bundle>.sha256sidecar is present) the outer SHA-256 of the.tar.gz.Internal checksums listed in
checksums.txt(per-file SHA-256).DSSE payload ↔ statement consistency — the base64 DSSE payload is decoded and its in-toto
_typefield must matchstatement.json’s_type.Transparency presence — when not offline,
transparency.ndjsonmust be present in the bundle.Important limitation (from
VerifyDsseSignature): in offline mode the verifier explicitly skips the cryptographic signature check (“would require access to signing keys/certificates”). In online mode it currently only confirms the payload matches the statement and that ≥1 signature entry exists — full trust-root signature validation is not yet wired into the CLI verifier.
Proof Spine (API) — ProofSpineVerifier checks each segment and the spine:
- Index continuity (
segment.Index == position). - Prev-hash chaining (segment n’s
prevSegmentHashequals segment n-1’sresultHash; the first segment’s prev hash must be null). - Segment-id integrity —
segmentIdis recomputed fromtype:index:inputHash:resultHash:prevHashand compared. - DSSE signature per segment (HMAC via
HmacDsseSigningService), yielding a per-segment status ofverified,untrusted, orinvalid. - Signed-payload field consistency — the signed segment payload’s
segmentType/index/inputHash/resultHash/prevSegmentHashmust match the segment record. - Root-hash and spine-id integrity — the spine
rootHashis recomputed as a hash over the colon-joined segmentresultHashvalues, andspineIdis recomputed fromartifactId:vulnerabilityId:policyProfileId:rootHash.
Verification Components
| Component | Where | Verification |
|---|---|---|
Attestation DSSE envelope (attestation.dsse.json) | CLI bundle | Payload ↔ statement _type; signature presence |
in-toto statement (statement.json) | CLI bundle | Subjects + predicate type extraction |
checksums.txt / <bundle>.sha256 | CLI bundle | SHA-256 integrity |
transparency.ndjson | CLI bundle | Presence (online); content not re-verified by the CLI |
| Proof spine segments | Scanner API | DSSE (HMAC) + hash-chain + id recomputation |
| Spine root hash | Scanner API | Recompute from segment result hashes |
Trust Model (as implemented)
in-toto Statement (statement.json)
└── DSSE Envelope (attestation.dsse.json) ← payload _type must match statement
└── Attestation bundle (.tar.gz) ← checksums.txt + optional .sha256
└── transparency.ndjson (optional Rekor entries)
Proof Spine (Scanner)
Segment[0] SBOM_SLICE ──prevHash=null
Segment[1] MATCH ──prevHash=Segment[0].resultHash
Segment[2] REACHABILITY──prevHash=Segment[1].resultHash
... each DSSE-signed (HMAC)
rootHash = hash( join(":", segment.resultHash[]) )
spineId = hash( artifactId:vulnId:profileId:rootHash )[0..32]
Note on “Merkle”.
ProofSpineModelsdescribesRootHashas a “Merkle root hash”, butProofSpineVerifier.ComputeRootHashis a flat hash of the colon-joined segment result hashes — not a binary Merkle tree with inclusion proofs. There is no X.509 root/intermediate CA hierarchy in the proof code; DSSE segments are signed with an HMAC key profile (HmacDsseSigningService).
2. Attestation Bundle Verification (CLI)
2.1 Basic Bundle Verification
The stella proof verify command operates on an attestation bundle .tar.gz (see §2.4 for the layout). Source: ProofCommandGroup.BuildVerifyCommand → AttestationBundleVerifier.VerifyAsync.
# Verify an attestation bundle (.tar.gz)
stella proof verify --bundle bundle.tar.gz
# Short flag form
stella proof verify -b bundle.tar.gz
# Offline mode (skips Rekor/transparency *and* the crypto signature check)
stella proof verify --bundle bundle.tar.gz --offline
# Machine-readable output
stella proof verify --bundle bundle.tar.gz --output json
Flags implemented by the command:
| Flag | Alias | Effect |
|---|---|---|
--bundle | -b | Path to the attestation bundle .tar.gz (required). |
--offline | — | Skip transparency check and crypto signature verification. |
--output | -o | text (default) or json. |
--verbose | — | Enables debug logging only; does not add output sections. |
Not implemented.
--check-rekor,--skip-rekor, and--crlare not options of this command. Online verification implies transparency-presence checking;--offlineis the only modifier. There is no separatestella proof compute-root,stella proof rekor-entry,stella proof show, orstella proof offline-kitverb (see §5 and the CLI quick reference).
Expected Output (Success — text)
PrintTextResult emits:
Proof Verification Result
========================================
Status: PASS
Bundle: bundle.tar.gz
Root Hash: sha256:abc123...
Attestation ID: <id from metadata.json>
Export ID: <export id from metadata.json>
Predicate: <predicateType from statement.json>
Subjects: 1
- my-image@sha256:...
Verification Checks:
----------------------------------------
[PASS] File integrity
[PASS] DSSE envelope format
[PASS] Signature validation
[PASS] Transparency log (or [SKIP] Transparency log (offline mode))
Expected Output (Success — json)
{
"valid": true,
"status": "verified",
"bundlePath": "bundle.tar.gz",
"rootHash": "sha256:abc123...",
"attestationId": "...",
"exportId": "...",
"subjects": ["my-image@sha256:..."],
"predicateType": "...",
"checks": [
{"check": "file_integrity", "status": "pass", "details": "Bundle checksums verified"},
{"check": "dsse_signature", "status": "pass", "details": "DSSE envelope signature valid"},
{"check": "transparency_log", "status": "pass", "details": "Transparency entry verified or skipped (offline)"}
]
}
The check names emitted by
BuildVerificationChecksare exactlyfile_integrity,dsse_signature, andtransparency_log. There are nomerkle_root,certificate_chain, ornot_expiredchecks.
Expected Output (Failure)
On failure the text renderer prints Status: FAIL and a single [FAIL] <error message> line; the json renderer sets valid: false and a status of failed (or error for missing file / exceptions). The process exit code is set from AttestationBundleExitCodes (see §2.3).
2.2 Verification Check Order
AttestationBundleVerifier.VerifyAsync runs these steps in order; the first failure short-circuits with the corresponding exit code:
- Bundle file exists →
FileNotFound (6). - Outer SHA-256 against a co-located
<bundle>.tar.gz.sha256sidecar if present (skipped silently when absent) →ChecksumMismatch (2). - Extract the gzip+tar archive →
FormatError (5)on archive/JSON errors. - Internal checksums from
checksums.txt(transparency.ndjsontreated as optional) →ChecksumMismatch (2). - DSSE signature — decode the base64 payload and require its in-toto
_typeto equalstatement.json’s_type; online mode additionally requires ≥1 signature entry →SignatureFailure (3). - Transparency presence — when not offline, require
transparency.ndjson→MissingTransparency (4).
2.3 Exit Codes
AttestationBundleExitCodes (src/Cli/StellaOps.Cli/Services/Models/AttestationBundleModels.cs):
| Code | Constant | Meaning |
|---|---|---|
| 0 | Success | Bundle verified. |
| 1 | GeneralFailure | General failure. |
| 2 | ChecksumMismatch | Outer or internal checksum mismatch. |
| 3 | SignatureFailure | DSSE payload/statement mismatch or missing signature. |
| 4 | MissingTransparency | transparency.ndjson missing (online mode). |
| 5 | FormatError | Archive or file-format error. |
| 6 | FileNotFound | Bundle file not found. |
| 7 | ImportFailed | (Import path only.) |
The unrelated
ProofExitCodesclass (Success/PolicyViolation/SystemError/…) is used for the command-level wrapper and error paths inProofCommandGroup; the bundle-verification result returnsAttestationBundleExitCodes.
2.4 Attestation Bundle Layout
The .tar.gz produced by the Export Center and consumed by the CLI contains these entries (AttestationBundleVerifier constants):
bundle.tar.gz
├── attestation.dsse.json # DSSE envelope { payloadType, payload(b64), signatures[] }
├── statement.json # in-toto statement (_type, predicateType, subject[])
├── transparency.ndjson # optional Rekor entries (one JSON object per line)
├── metadata.json # exportId, attestationId, tenantId, rootHash, subjectDigests[]
└── checksums.txt # "<sha256> <filename>" lines (optional, '#'-comments allowed)
bundle.tar.gz.sha256 # optional co-located outer-checksum sidecar
This is not the
manifest.json/certificate.pem/dsse-envelope.jsonlayout described in earlier drafts. There is nocertificate.pemin the bundle; trust is anchored via the DSSE envelope and (optionally) transparency entries, not an embedded X.509 chain.
2.5 Inspecting a Bundle Manually
# List bundle contents
tar -tzf bundle.tar.gz
# Inspect the DSSE envelope
tar -xzOf bundle.tar.gz attestation.dsse.json | jq .
# Decode the DSSE payload (the in-toto statement)
tar -xzOf bundle.tar.gz attestation.dsse.json | jq -r '.payload' | base64 -d | jq .
# Confirm payload _type matches the statement
tar -xzOf bundle.tar.gz statement.json | jq -r '._type'
# Verify the outer checksum (if a .sha256 sidecar exists)
sha256sum -c bundle.tar.gz.sha256
3. Proof Spine Verification (API)
A proof spine is the Scanner’s hash-chained evidence record for one artifact/vulnerability/policy-profile tuple. Endpoints are mapped by ProofSpineEndpoints.MapProofSpineEndpoints (read-only; both require the scanner.scans.read scope) and verified by ProofSpineVerifier.
3.1 Endpoints
| Method | Path (default segments) | Returns | Scope |
|---|---|---|---|
GET | …/spines/{spineId} | Full spine + per-segment + spine-level verification | scanner.scans.read |
GET | …/scans/{scanId}/spines | Summaries of spines for a scan run | scanner.scans.read |
Path prefixes come from
ScannerWebServiceOptions.Api(SpinesSegmentdefaultspines,ScansSegmentdefaultscans) under the configured API group. Both endpoints also support CBOR viaAccept: application/cbor. There is noPOST .../proofs/{rootHash}/verifyendpoint — verification runs server-side onGET .../spines/{spineId}and is returned inline.Scope note (flag).
scanner.scans.readis declared locally in the Scanner WebService (ScannerAuthorityScopes/ScannerPolicies) and is not present in the canonical Authority scope catalog (StellaOps.Auth.Abstractions/StellaOpsScopes.cs, which uses dotted scopes likescanner:read/scanner:export). Verify the token actually carriesscanner.scans.read(and that it is provisioned in the client’sallowed_scopes) before relying on these endpoints in a given environment.
# Fetch and verify a single spine
curl "https://scanner.stellaops.local/<api>/spines/$SPINE_ID" \
-H "Authorization: Bearer $TOKEN"
# List spines for a scan run
curl "https://scanner.stellaops.local/<api>/scans/$SCAN_ID/spines" \
-H "Authorization: Bearer $TOKEN"
3.2 Spine Response Shape
The single-spine response (ProofSpineResponseDto) includes the segments and an inline verification block. Each segment carries its own DSSE envelope, the hash-chain fields, and (on failure) verificationErrors:
{
"spineId": "…",
"artifactId": "…",
"vulnerabilityId": "…",
"policyProfileId": "…",
"verdict": "…",
"verdictReason": "…",
"rootHash": "…",
"scanRunId": "…",
"supersededBySpineId": null,
"segments": [
{
"segmentId": "…",
"segmentType": "SBOM_SLICE",
"index": 0,
"inputHash": "…",
"resultHash": "…",
"prevSegmentHash": null,
"envelope": { "payloadType": "application/vnd.stellaops.proofspine.segment+json",
"payload": "<b64>", "signatures": [{ "keyid": "…", "sig": "…" }] },
"toolId": "…",
"toolVersion": "…",
"status": "verified",
"createdAt": "…",
"verificationErrors": null
}
],
"verification": { "isValid": true, "errors": [] }
}
Segment segmentType wire values: SBOM_SLICE, MATCH, REACHABILITY, GUARD_ANALYSIS, RUNTIME_OBSERVATION, POLICY_EVAL.
Per-segment status (lowercased from ProofSegmentStatus): verified, untrusted (signature valid but key not trusted), or invalid.
3.3 Spine-Level Verification Errors
ProofSpineVerifier records these spine-level error codes in verification.errors (and per-segment in verificationErrors):
| Error code | Meaning |
|---|---|
root_hash_mismatch | Recomputed spine root hash ≠ stored rootHash. |
spine_id_mismatch | Recomputed spineId ≠ stored spineId. |
segment_index_mismatch:<a>-><b> | Segment index not equal to its position. |
prev_hash_expected_null | First segment has a non-null prevSegmentHash. |
prev_hash_mismatch | Segment prev hash ≠ previous segment resultHash. |
segment_id_mismatch | Recomputed segment id ≠ stored segmentId. |
signed_segment_type_mismatch / signed_index_mismatch / signed_fields_mismatch | Signed payload disagrees with the segment record. |
dsse_payload_not_base64 / signed_payload_json_invalid | Malformed signed payload. |
<dsse failure reason> | Whatever HmacDsseSigningService.VerifyAsync reports (e.g. dsse_key_not_trusted). |
A spine is isValid: true only when there are no spine-level errors and every segment status is verified or untrusted.
Defense in depth.
HandleGetSpineAsyncre-validates the assembled DTO throughProofSegmentValidatorand records structural drift to thescanner_proofspine_validation_failures_totalcounter. This validation does not gate the HTTP response.
3.4 CLI Spine Command (partial)
stella proof spine show <bundle-id> exists but is not yet implemented — it prints a placeholder (“Spine display not yet implemented”) and directs the user to stella proof verify. There is no stella proof spine --bundle form. Use the HTTP API above for spine inspection until the CLI verb lands.
4. Offline Verification
4.1 When to Use Offline Verification
- Air-gapped environments
- Network-restricted systems
- Compliance audits without API access
- Disaster recovery scenarios
4.2 Prerequisites
For attestation-bundle verification, the only required input is the bundle .tar.gz itself; an optional co-located <bundle>.tar.gz.sha256 sidecar lets the verifier check the outer checksum. The DSSE envelope, in-toto statement, internal checksums, and (optional) transparency entries are all carried inside the bundle, so no external CA bundle or trust-root file is consumed by the CLI verifier today.
Not implemented. There is no
stella proof offline-kit createcommand and noverify.sh/ca-bundle.pem/trust-roots.jsonkit generator. The--offline-kitflag that exists in the CLI belongs to thestella sbomer drift analyzecommand (aliasdiff), not to proof verification. (Thestella offline kit …command tree is a separate pull/import/status surface for offline kits, unrelated to proof verification.)
4.3 Running Offline Verification
# Offline bundle verification
stella proof verify --bundle bundle.tar.gz --offline
4.4 Offline Verification Checks (as implemented)
| Check | Online | Offline | Notes |
|---|---|---|---|
| Bundle exists | ✓ | ✓ | Local file check. |
Outer SHA-256 (.sha256 sidecar) | ✓ | ✓ | Only if sidecar present. |
Internal checksums (checksums.txt) | ✓ | ✓ | Per-file SHA-256. |
DSSE payload ↔ statement _type | ✓ | ✓ | Local JSON comparison. |
| DSSE cryptographic signature | partial¹ | ✗ | Skipped entirely offline. |
Transparency (transparency.ndjson) presence | ✓ | ✗ | Required only online. |
¹ Online mode currently only checks that ≥1 signature entry exists and that the payload matches the statement; full trust-root signature validation is not yet wired into AttestationBundleVerifier. Flag for follow-up.
4.5 Air-Gap Considerations
Because the bundle is self-contained, air-gap transfer is just a matter of moving the .tar.gz (and its optional .sha256 sidecar) across the boundary and running stella proof verify --bundle … --offline. There is no stella ca export command and no CRL/OCSP support in the proof CLI; certificate revocation checking is not implemented.
5. Transparency Log Integration
5.1 Rekor Overview
StellaOps can publish attestations to a Sigstore-style Rekor transparency log, and the Attestor service periodically re-verifies inclusion (RekorVerificationJob / RekorVerificationService, sprint SPRINT_20260117_001_ATTESTOR_periodic_rekor_verification). Within the proof surfaces covered by this runbook, transparency manifests as the transparency.ndjson entry inside an attestation bundle.
in-toto statement → DSSE envelope → (optional) Rekor → transparency.ndjson in bundle
5.2 Transparency in the CLI Verifier
The CLI verifier treats transparency as a presence check, not a re-validation:
- Online (
stella proof verify --bundle …): the bundle must containtransparency.ndjson, otherwise the command fails withMissingTransparency (4). The verifier does not re-query Rekor or recompute the inclusion proof. - Offline (
--offline): the transparency check is skipped and the text output shows[SKIP] Transparency log (offline mode).
# Inspect the transparency entries shipped in the bundle
tar -xzOf bundle.tar.gz transparency.ndjson | jq .
Not implemented.
stella proof verify --check-rekor,stella proof verify --skip-rekor, andstella proof rekor-entryare not CLI verbs/flags. There is no built-inrekor-cliwrapper instella proof.
5.3 Attestor-Side Rekor Verification
Continuous Rekor inclusion verification is an Attestor-service concern, surfaced operationally through:
- Attestor metrics:
attestor_rekor_verification_runs_total,attestor_rekor_verification_run_failures_total,attestor_rekor_verification_failure_rate. - Doctor checks:
RekorConnectivityCheck,RekorVerificationJobCheck,TransparencyLogConsistencyCheck,CosignKeyMaterialCheck(src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Attestor/).
See the Attestor module dossier for the Rekor verification job configuration (RekorVerificationOptions).
6. Troubleshooting
6.1 DSSE Signature / Payload Mismatch
Symptom (CLI): exit code 3 (SignatureFailure); message such as DSSE payload does not match statement _type, DSSE envelope not found or has no payload, or DSSE envelope has no signatures.
Diagnostic Steps:
# Inspect the envelope and statement
tar -xzOf bundle.tar.gz attestation.dsse.json | jq .
tar -xzOf bundle.tar.gz statement.json | jq -r '._type'
# Decode the payload and compare its _type
tar -xzOf bundle.tar.gz attestation.dsse.json | jq -r '.payload' | base64 -d | jq -r '._type'
Common Causes:
| Cause | Resolution |
|---|---|
| Bundle modified after export | Re-export from the Export Center. |
Payload/statement _type drift | Re-export; the envelope payload must match statement.json. |
Envelope has no signatures (online) | Re-export with signing enabled, or verify with --offline. |
| Base64/encoding corruption | Re-download; the payload must be valid base64. |
6.2 Spine Root-Hash / Chain Mismatch (API)
Symptom (API): verification.errors contains root_hash_mismatch, spine_id_mismatch, prev_hash_mismatch, or segment_id_mismatch; one or more segments report status: invalid.
Diagnostic Steps:
- Inspect the inline
verificationblock and each segment’sverificationErrorsfromGET …/spines/{spineId}. - Confirm segment ordering:
indexmust be contiguous from 0 and eachprevSegmentHashmust equal the previous segment’sresultHash. - Confirm the spine root hash: it is the hash of the colon-joined
resultHashvalues across all segments, in order.
Resolution:
- A mismatch means the persisted spine data has drifted from its signed form; re-run the scan that produced the spine, or re-ingest the evidence.
- Check the
scanner_proofspine_validation_failures_totalcounter to see whether the stored spine is structurally out of contract.
6.3 Checksum Mismatch (CLI)
Symptom: exit code 2 (ChecksumMismatch); message SHA-256 checksum mismatch (outer sidecar) or Checksum mismatch for '<file>' (internal).
Diagnostic Steps:
# Verify the outer checksum if a sidecar exists
sha256sum -c bundle.tar.gz.sha256
# Re-derive a single internal file's hash and compare to checksums.txt
tar -xzOf bundle.tar.gz attestation.dsse.json | sha256sum
tar -xzOf bundle.tar.gz checksums.txt
Resolution:
- Re-download/transfer the bundle (and its
.sha256sidecar) — a mismatch is almost always corruption in transit or post-export tampering.
6.4 Bundle Extraction / Format Error
Symptom: exit code 5 (FormatError); message Failed to extract bundle contents.
Diagnostic Steps:
file bundle.tar.gz
gzip -t bundle.tar.gz
tar -tzf bundle.tar.gz
ls -la bundle.tar.gz
Resolution:
- Re-download if corrupted or truncated.
- Confirm the archive is gzip+tar (the verifier uses
GZipStream+TarReader). - Verify sufficient disk space.
7. Monitoring & Alerting
Reconciliation note. Earlier drafts listed metrics such as
proof_verification_total,proof_verification_failures,proof_verification_duration_ms,proof_verification_success_total, andcertificate_expiry_days. None of those metric names exist insrc/. The metrics below are the ones actually emitted. The CLI verifier (stella proof verify) is a one-shot client command and does not export Prometheus metrics.
7.1 Key Metrics (as implemented)
| Metric | Source | Notes |
|---|---|---|
scanner_proofspine_validation_failures_total | ProofSpineValidationMetrics (meter StellaOps.Scanner.ProofSpine) | Incremented when a served spine DTO fails structural re-validation; tagged reasonCode + tenantId (tenantId defaults to unknown). |
scanner_proofspine_validation_successes_total | ProofSpineValidationMetrics (meter StellaOps.Scanner.ProofSpine) | Incremented on successful structural re-validation; tagged tenantId. Used to compute a validation failure rate alongside the failures counter. |
attestor_rekor_verification_runs_total | RekorVerificationMetrics (Attestor) | Count of periodic Rekor inclusion-verification runs. |
attestor_rekor_verification_run_failures_total | RekorVerificationMetrics (Attestor) | Count of failed Rekor verification runs. |
attestor_rekor_verification_failure_rate | RekorVerificationMetrics (Attestor) | Rolling failure rate of Rekor verification. |
The proof-spine validation metrics live in the ProofSpine library, not the WebService: confirm against
src/Scanner/__Libraries/StellaOps.Scanner.ProofSpine/Diagnostics/ProofSpineValidationMetrics.cs(meter nameStellaOps.Scanner.ProofSpine; countersFailureCounterName/SuccessCounterName) and the Rekor metrics againstsrc/Attestor/StellaOps.Attestor/StellaOps.Attestor.Core/Verification/RekorVerificationMetrics.csbefore wiring dashboards; do not rely on the metric names alone.
7.2 Example Grafana / Prometheus Queries
# Proof-spine structural validation failures (Scanner)
sum(rate(scanner_proofspine_validation_failures_total[1h]))
# Rekor verification failure rate (Attestor)
attestor_rekor_verification_failure_rate
# Failed Rekor verification runs
sum(rate(attestor_rekor_verification_run_failures_total[1h]))
7.3 Example Alert Rules
groups:
- name: proof-verification
rules:
- alert: ProofSpineValidationFailures
expr: rate(scanner_proofspine_validation_failures_total[1h]) > 0
for: 15m
labels:
severity: warning
annotations:
summary: Scanner is serving proof spines that fail structural re-validation
- alert: RekorVerificationFailing
expr: attestor_rekor_verification_failure_rate > 0
for: 1h
labels:
severity: warning
annotations:
summary: Attestor periodic Rekor inclusion verification is failing
There is no signing-certificate-expiry metric or alert in the proof code today; any certificate-lifecycle alerting belongs to the signing/KMS subsystem, not this runbook. Flag for follow-up if cert-expiry alerting is required.
8. Escalation Procedures
8.1 Escalation Matrix
| Severity | Condition | Response Time | Escalate To |
|---|---|---|---|
| P1 - Critical | Mass bundle/spine verification failures (tamper suspected) | 15 minutes | Platform Team + Security Team |
| P2 - High | Attestor Rekor verification failing (attestor_rekor_verification_failure_rate > 0) | 1 hour | Platform Team |
| P2 - High | Persistent scanner_proofspine_validation_failures_total | 1 hour | Scanner Team |
| P3 - Medium | Single bundle verification failure | 4 hours | Support Queue |
8.2 P1: Mass Verification Failures Response
Immediate Actions (0-15 min):
- Capture exit codes /
verification.errorsfrom failing verifications. - Determine whether the failure is integrity (
ChecksumMismatch,root_hash_mismatch) — indicating tamper — versus format/transport. - Notify stakeholders; treat suspected tamper as a security incident.
- Capture exit codes /
Triage (15-60 min):
- Re-export/re-fetch a known-good bundle or spine and re-verify to isolate whether the issue is producer-side or transport-side.
- For spine failures, check
scanner_proofspine_validation_failures_totaland the Doctor Attestor checks (RekorConnectivityCheck, etc.).
Post-Incident:
- Record root cause and remediation in the relevant sprint Execution Log.
- File follow-up work for any verification gaps surfaced (e.g. full trust-root signature validation in the CLI verifier — see §4.4).
8.3 Contacts
Update these to your deployment’s actual on-call rosters; the addresses below are placeholders, not verified routing.
| Role | Contact | Availability |
|---|---|---|
| Security Team | security@stellaops.io | Business hours |
| Platform On-Call | platform-oncall@stellaops.io | 24/7 |
| Attestor / Scanner Team | (per deployment) | Business hours |
Appendix A: DSSE Envelope & Payload Types
A.1 Attestation-bundle DSSE envelope (attestation.dsse.json)
{
"payloadType": "<in-toto predicate / statement media type>",
"payload": "<base64-encoded in-toto statement>",
"signatures": [
{ "keyid": "<key id>", "sig": "<base64-encoded signature>" }
]
}
The base64 payload decodes to the in-toto statement.json (with _type, predicateType, and a subject[] array). The CLI verifier requires the decoded payload’s _type to equal statement.json’s _type. Earlier drafts that claimed a manifest predicate with sbomHash/rulesHash/policyHash/feedHash and a fixed payloadType of application/vnd.stellaops.proof+json do not match the code — there is no such fixed payload type and the manifest hash fields are named concelierSnapshotHash, excititorSnapshotHash, and latticePolicyHash on ScanManifest.
A.2 Proof-spine segment DSSE envelope
Each spine segment carries a DSSE envelope whose payload type is application/vnd.stellaops.proofspine.segment+json (ProofSpineBuilder.DefaultSegmentPayloadType). The decoded payload contains segmentType, index, inputHash, resultHash, and prevSegmentHash, which the verifier cross-checks against the segment record.
A.3 Function-proof (FuncProof) DSSE envelope
FuncProof documents use the media type application/vnd.stellaops.funcproof+json (FuncProofConstants.MediaType, and the same value for FuncProofConstants.DssePayloadType). The DSSE envelope is verified by FuncProofDsseService (src/Scanner/__Libraries/StellaOps.Scanner.Evidence/FuncProofDsseService.cs).
Flag — verbs not wired. The
stella funcproof generate|verify|info|exportcommand builders and handlers exist (src/Cli/StellaOps.Cli/Commands/Proof/FuncProofCommandGroup.cs+FuncProofCommandHandlers.cs), butFuncProofCommandGroup.BuildFuncProofCommandis never registered into the CLI root —CommandFactory.BuildRootCommandwiresProofCommandGroup.BuildProofCommand(CommandFactory.cs:159) but has no correspondingBuildFuncProofCommandcall. As a resultstella funcproof …is not invokable in the current build despite the code being present. See the Appendix B note before documenting these as available.
A.4 Score proof bundle (ProofBundleWriter)
The score replay/proof bundle written by ProofBundleWriter is a ZIP. The current CreateZipBundleAsync writes exactly four entries: manifest.json, manifest.dsse.json, score_proof.json (the ProofLedger nodes + rootHash, serialized via ProofLedger.ToJson), and meta.json. The ledger rootHash (ProofHashing.ComputeRootHash) is sha256:<hex> over the canonical JSON array of per-node hashes — a flat ordered chain, not a Merkle tree.
Flag. The
ProofBundleWriterclass XML doc-comment lists an optionalproof_root.dsse.json(DSSE envelope for the root hash), but the currentCreateZipBundleAsyncdoes not emit it andReadBundleAsyncdoes not read it — it is aspirational, not produced today. Do not expect aproof_root.dsse.jsonmember in bundles from this writer.
Appendix B: CLI Quick Reference
# Attestation-bundle verification (ProofCommandGroup)
stella proof verify --bundle <path.tar.gz> # Verify bundle (alias -b)
stella proof verify --bundle <path> --offline # Skip transparency + crypto sig check
stella proof verify --bundle <path> --output json # JSON output (alias -o; text|json)
stella proof verify --bundle <path> --verbose # Extra debug logging only
# Proof-spine (CLI verb exists but is NOT YET IMPLEMENTED)
stella proof spine show <bundle-id> # Prints placeholder; use the HTTP API
# Function-level proof (FuncProofCommandGroup) — BUILDERS EXIST BUT NOT WIRED INTO THE CLI ROOT (see below)
stella funcproof generate --binary <path> [--sign] [--transparency] [--registry <ref>]
stella funcproof verify --proof <path> [--binary <path>] [--offline] [--strict] [--format text|json]
stella funcproof info <proof>
stella funcproof export <proof> --output <dir> [--format bundle|evidence-locker] [--include dsse,tlog-receipt,raw-proof]
Flag —
stella funcproofis not currently reachable. The flag/verb shapes above are accurate toFuncProofCommandGroup, butBuildFuncProofCommandis never added to the root command (CommandFactory.BuildRootCommandregistersBuildProofCommandonly). Until aroot.Add(FuncProofCommandGroup.BuildFuncProofCommand(...))registration lands, treatstella funcproof …as not yet available at the CLI even though the implementation exists. (scanner.funcproof*keys inConfigCatalogare server-side scanner config, not CLI command registrations.)
Not implemented (do not document as available):
stella proof verify --check-rekor/--skip-rekor/--crl,stella proof compute-root,stella proof rekor-entry,stella proof show,stella proof offline-kit …, andstella signer cert show|rotate|export/stella ca export. The--offline-kitflag exists only onsbomer drift analyze(aliasdiff).
Revision History
| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0.0 | 2025-12-20 | Agent | Initial release |
| 1.1.0 | 2026-05-30 | Agent | Deep doc↔code reconciliation against src/Cli (ProofCommandGroup, AttestationBundleVerifier, FuncProofCommandGroup) and src/Scanner (ProofSpineEndpoints, ProofSpineVerifier, ProofBundleWriter). Corrected bundle layout (.tar.gz with attestation.dsse.json/statement.json/transparency.ndjson/metadata.json/checksums.txt), check names (file_integrity/dsse_signature/transparency_log), CLI flags/verbs, exit codes, DSSE payload types, ScanManifest hash fields, real metric names, and the API spine endpoints/scope (scanner.scans.read). Removed fabricated Merkle-tree / X.509 cert-chain / POST …/verify / offline-kit / stella signer cert content and flagged the offline crypto-signature gap. |
| 1.2.0 | 2026-05-31 | Agent | Second reconciliation pass. Fixed ProofSpineValidationMetrics source location (ProofSpine library Diagnostics, not WebService) and footnote path; added the companion scanner_proofspine_validation_successes_total counter and the real meter name (StellaOps.Scanner.ProofSpine) + tags (reasonCode/tenantId). Corrected Appendix A.4 — ProofBundleWriter.CreateZipBundleAsync emits only manifest.json/manifest.dsse.json/score_proof.json/meta.json; proof_root.dsse.json is an aspirational doc-comment entry that is not produced. Corrected --offline-kit ownership (sbomer drift analyze / alias diff, not scan drift analyze). Flagged that stella funcproof … builders exist but BuildFuncProofCommand is never registered into the CLI root (CommandFactory.cs:159 wires only BuildProofCommand), so the verbs are not currently invokable. |
