checkId: check.compliance.evidence-rate plugin: stellaops.doctor.compliance severity: fail tags: [compliance, evidence, attestation]

Evidence Generation Rate

Doctor check check.compliance.evidence-rate — for operators watching that Stella Ops is reliably recording evidence for every release decision, scan, and policy evaluation.

What It Checks

Monitors evidence generation success rate by querying the Evidence Locker at /api/v1/evidence/metrics. The check computes the success rate as (totalGenerated - failed) / totalGenerated over the last 24 hours and compares it against two thresholds:

Evidence collected: success_rate, total_generated_24h, failed_24h, pending_24h, avg_generation_time_ms.

The check only runs when EvidenceLocker:Url or Services:EvidenceLocker:Url is configured. It uses a 10-second HTTP timeout. If no evidence has been generated (totalGenerated == 0), the success rate defaults to 100%.

Why It Matters

Evidence generation is a critical path in the release pipeline. Every release decision, scan result, and policy evaluation produces evidence that feeds compliance audits and attestation chains. A dropping success rate means evidence records are being lost, which creates gaps in the audit trail. Below 95%, the system is losing more than 1 in 20 evidence records, making compliance reporting unreliable and potentially invalidating release approvals that lack supporting evidence.

Common Causes

• Evidence generation service failures (internal errors, OOM)
• Database connectivity issues preventing evidence persistence
• Signing key unavailable, blocking signed evidence creation
• Storage quota exceeded on the evidence backend
• Intermittent failures due to high load or resource contention

How to Fix

Docker Compose

# Check evidence locker logs for errors
docker compose logs evidence-locker --since 1h | grep -i error

# Verify signing keys
docker compose exec evidence-locker stella evidence keys status

# Check database connectivity
docker compose exec evidence-locker stella evidence db check

# Check storage capacity
docker compose exec evidence-locker df -h /data/evidence

# If storage is full, clean up or expand volume
docker compose exec evidence-locker stella evidence cleanup --older-than 90d --dry-run

Bare Metal / systemd

# Check service logs
journalctl -u stellaops-evidence-locker --since "1 hour ago" | grep -i error

# Verify signing keys
stella evidence keys status

# Check database connectivity
stella evidence db check

# Check storage usage
df -h /var/lib/stellaops/evidence

sudo systemctl restart stellaops-evidence-locker

Kubernetes / Helm

# Check evidence locker pod logs
kubectl logs deploy/stellaops-evidence-locker --since=1h | grep -i error

# Verify signing keys
kubectl exec deploy/stellaops-evidence-locker -- stella evidence keys status

# Check persistent volume usage
kubectl exec deploy/stellaops-evidence-locker -- df -h /data/evidence

# Check for OOMKilled pods
kubectl get events --field-selector reason=OOMKilled -n stellaops

Verification

stella doctor run --check check.compliance.evidence-rate

Related Checks

• check.compliance.attestation-signing — signing key health affects evidence generation
• check.compliance.evidence-integrity — integrity of generated evidence
• check.compliance.provenance-completeness — provenance depends on evidence generation
• check.compliance.audit-readiness — overall audit readiness depends on evidence availability

See the Doctor reference for the full check catalog, CLI usage, and export bundles.