checkId: check.evidencelocker.provenance plugin: stellaops.doctor.evidencelocker severity: fail tags: [evidence, provenance, integrity, chain]

Provenance Chain Integrity

This stella doctor check spot-checks the integrity of stored provenance records by recomputing their content hashes, so operators can detect corruption or tampering that would invalidate the supply-chain guarantee for a release.

What It Checks

Validates provenance chain integrity using random sample verification. The check operates on provenance records stored in the provenance/ subdirectory of the evidence locker path. It performs:

  1. Random sampling: selects up to 5 random provenance records from the pool for validation (configurable via SampleSize constant).
  2. Hash verification: for each sampled record, reads the contentHash and payload fields, recomputes the SHA-256 hash of the payload, and compares it against the declared hash. Supports sha256: prefixed hash values.
  3. Structural validation: verifies that each record contains required contentHash and payload fields.
ConditionResult
Evidence locker path not configuredSkip
No provenance directory or no recordsPass (nothing to verify)
Any sampled records fail hash verificationFail
All sampled records pass hash verificationPass

Evidence collected: TotalRecords, SamplesChecked, ValidCount, InvalidCount, InvalidRecords.

The check only runs when EvidenceLocker:Path is configured and the directory exists.

Why It Matters

Provenance records link each software artifact to its build source, build system, and build steps. The content hash ensures that the provenance payload has not been modified since it was created. A broken hash indicates the provenance record was corrupted or tampered with, which invalidates the supply-chain integrity guarantee for the associated release. Even a single invalid provenance record undermines trust in the entire provenance chain and should be investigated as a potential security incident.

Common Causes

How to Fix

Docker Compose

# Run full provenance audit
docker compose exec evidence-locker stella evidence audit --type provenance --full

# Check specific invalid records
docker compose exec evidence-locker stella evidence verify --id <record-id>

# Review evidence locker integrity
docker compose exec evidence-locker stella evidence integrity-check

# Check for storage errors
docker compose exec evidence-locker dmesg | grep -i error

# Check disk health
docker compose exec evidence-locker df -h /data/evidence/provenance/

Bare Metal / systemd

# Full provenance audit
stella evidence audit --type provenance --full

# Verify specific records
stella evidence verify --id <record-id>

# Full integrity check
stella evidence integrity-check

# Check filesystem health
sudo fsck -n /dev/sda1

# Check for disk I/O errors
dmesg | grep -i "i/o error"

# List provenance records
ls -la /var/lib/stellaops/evidence/provenance/

Kubernetes / Helm

# Full provenance audit
kubectl exec deploy/stellaops-evidence-locker -- stella evidence audit --type provenance --full

# Verify specific record
kubectl exec deploy/stellaops-evidence-locker -- stella evidence verify --id <record-id>

# Integrity check
kubectl exec deploy/stellaops-evidence-locker -- stella evidence integrity-check

# Check persistent volume health
kubectl describe pvc stellaops-evidence-data

# Check for pod restarts that might indicate storage issues
kubectl get events --field-selector involvedObject.name=stellaops-evidence-locker -n stellaops

# values.yaml - schedule periodic integrity checks
evidenceLocker:
  integrityCheck:
    enabled: true
    schedule: "0 4 * * *"  # Daily at 4am
    sampleSize: 10

Verification

stella doctor run --check check.evidencelocker.provenance