checkId: check.attestation.rekor.verification.job plugin: stellaops.doctor.attestor severity: warn tags: [attestation, rekor, verification, background]

Rekor Verification Job

This Stella Ops Doctor check reports the health of the background job that continuously re-verifies attestation entries against Rekor – the integrity watchdog for the attestation pipeline. It is part of the attestation diagnostics surfaced by stella doctor; see the Doctor overview for how checks are run and reported.

What It Checks

Monitors the health of the periodic background job that re-verifies attestation entries stored in Rekor. The check queries IRekorVerificationStatusProvider from DI and evaluates several conditions in priority order:

1. Service not registered: Skip if IRekorVerificationStatusProvider is not in the DI container.
2. Never run: Warn if LastRunAt is null (job has never executed).
3. Critical alerts: Fail if CriticalAlertCount > 0 (possible log tampering, root hash mismatch, mass signature failures).
4. Root consistency failed: Fail if RootConsistent is false (stored checkpoint disagrees with remote log state).
5. Stale run: Warn if the job has not run in more than 48 hours.
6. High failure rate: Warn if FailureRate > 10% (more than 10% of verified entries failed).
7. Healthy: Pass with last run time, status, entries verified, failure rate, root consistency, and duration.

The check only runs when verification is enabled (Attestor:Verification:Enabled or Transparency:Verification:Enabled is not set to false).

Evidence collected: LastRun, LastRunStatus, IsRunning, NextScheduledRun, CriticalAlerts, RootConsistent, LastConsistencyCheck, HoursSinceLastRun, EntriesVerified, EntriesFailed, FailureRate, TimeSkewViolations, Duration.

Why It Matters

The verification job is the integrity watchdog for the attestation pipeline. It periodically re-checks that Rekor log entries have not been tampered with, that the root hash is consistent, and that signatures remain valid. Without this job running, an attacker could modify transparency log entries without detection, undermining the entire attestation trust model. A high failure rate may indicate clock skew, key rotation issues, or data corruption.

Common Causes

• Job was just deployed and has not run yet (first-run delay)
• Job is disabled in configuration
• Background service failed to start (DI error, missing dependency)
• Transparency log tampering detected (critical alert)
• Root hash mismatch with stored checkpoints
• Mass signature verification failures after key rotation
• Background service stopped or scheduler not running (stale run)
• Job stuck or failed repeatedly
• Clock skew causing timestamp validation failures (high failure rate)
• Invalid signatures from previous key rotations
• Corrupted entries in local database

How to Fix

Docker Compose

# Check attestor container status
docker compose -f devops/compose/docker-compose.stella-ops.yml ps attestor

# View verification job logs
docker compose -f devops/compose/docker-compose.stella-ops.yml logs attestor --tail 300 | \
  grep -i 'verification\|rekor'

# Trigger manual verification run
docker compose -f devops/compose/docker-compose.stella-ops.yml exec attestor \
  stella attestor verification run --now

# Enable verification if disabled
# ATTESTOR__VERIFICATION__ENABLED=true

# Restart attestor service
docker compose -f devops/compose/docker-compose.stella-ops.yml restart attestor

Bare Metal / systemd

# Check if the job is scheduled
stella attestor verification status

# Trigger a manual verification run
stella attestor verification run --now

# Check application logs for errors
journalctl -u stellaops-attestor --since '1 hour ago' | grep -i 'verification\|rekor'

# Review critical alerts
stella attestor verification alerts --severity critical

# Check transparency log status
stella attestor transparency status

# Review failed entries (high failure rate)
stella attestor verification failures --last-run

# Check system clock synchronization (if time skew violations)
timedatectl status

# Re-sync failed entries from Rekor
stella attestor verification resync --failed-only

# Restart the service if job is stale
sudo systemctl restart stellaops-attestor

# Review recent error logs (stale job)
journalctl -u stellaops-attestor --since '48 hours ago' | grep -i error

Kubernetes / Helm

# Check attestor pod status
kubectl get pods -l app.kubernetes.io/component=attestor -n stellaops

# View verification logs
kubectl logs -l app.kubernetes.io/component=attestor -n stellaops --tail=300 | \
  grep -i 'verification\|rekor'

# Trigger manual verification
kubectl exec -it deploy/stellaops-attestor -n stellaops -- \
  stella attestor verification run --now

# Enable verification in Helm values
# attestor:
#   verification:
#     enabled: true
#     intervalHours: 24
helm upgrade stellaops stellaops/stellaops -f values.yaml

# Restart attestor pods
kubectl rollout restart deployment/stellaops-attestor -n stellaops

If critical alerts indicate possible log tampering, this may be a security incident. Review evidence carefully before dismissing alerts.

Verification

stella doctor run --check check.attestation.rekor.verification.job

Related Checks

• check.attestation.rekor.connectivity – verification job requires Rekor connectivity
• check.attestation.transparency.consistency – complementary consistency check against stored checkpoints
• check.attestation.clock.skew – clock skew causes verification timestamp failures
• check.attestation.keymaterial – expired signing keys cause verification failures