Checkpoint Divergence Detection and Incident Response

Audience: Stella Ops platform operators and security/incident responders working with the Attestor service and Rekor transparency logs. Purpose: Explain how Stella Ops detects Rekor checkpoint divergence (split-view, rollback, cross-log, and staleness anomalies), what each alert and metric means, and how to respond when divergence is reported.

This runbook covers Rekor checkpoint divergence detection, anomaly types, alert payloads, metrics, and incident-response procedures.

Implementation status (verify against src/). The detection logic described here lives in StellaOps.Attestor.Core.Rekor.CheckpointDivergenceDetector and the alerting logic in CheckpointDivergenceAlertPublisher (project src/Attestor/StellaOps.Attestor/StellaOps.Attestor.Core/Rekor/). As of this writing those types are library code exercised only by unit tests — they are not registered in dependency injection, not invoked by any background service, and not surfaced through any HTTP endpoint or CLI command.

The entire IRekorCheckpointStore checkpoint-sync pipeline is library/test-only: RekorSyncBackgroundService (the periodic checkpoint/tile sync loop in StellaOps.Attestor.Core/Rekor/) is not registered as a hosted service in AttestorWebServiceComposition and IRekorCheckpointStore/PostgresRekorCheckpointStore are not registered by AddAttestorRuntimePersistence. The Attestor host’s only live external-transparency worker is TransparencySyncWorker(StellaOps.Attestor.Infrastructure/Workers/, gated behind attestor:Transparency:ExternalSync:Enabled + WorkerEnabled), which runs through a separate ITransparencySyncRunner path and does not populate the checkpoint store or call the divergence detector. So nothing in the running service fetches or stores StoredCheckpoint rows today, and nothing invokes the detector.

Treat the detection rules, metrics, and alert payloads below as the designed behaviour of the detector library; the live wiring (checkpoint-store registration, background invocation, metric export, Notify integration, operator tooling) is roadmap work. Sections that depend on tooling that does not yet exist are marked NOT IMPLEMENTED inline.

Overview

The checkpoint divergence detector (CheckpointDivergenceDetector) monitors the integrity of Rekor transparency logs by:

Divergence can indicate:

Detection Rules

The detector emits CheckpointAnomaly records with an AnomalyType and AnomalySeverity. The overall RecommendedAction (DivergenceAction) for a detection run is derived from the set of anomalies by CheckpointDivergenceDetector.DetermineAction: any root-hash mismatch ⇒ QuarantineAndAlert; otherwise any rollback ⇒ RejectAndAlert; otherwise the action follows the max severity (Critical ⇒ RejectAndAlert, Error/Warning ⇒ Alert, Info ⇒ Log, none ⇒ None). In practice the only Critical anomalies the detector raises are RootHashMismatch and TreeSizeRollback, both of which are caught by the earlier mismatch/rollback branches, so the Critical ⇒ RejectAndAlert arm of the severity switch is currently unreachable — it remains in source as a defensive fallback.

AnomalyTypeCondition (source)AnomalySeverityResulting DivergenceAction
RootHashMismatchStored checkpoint exists at same TreeSize with a different RootHash (DetectDivergenceAsync)CriticalQuarantineAndAlert
TreeSizeRollbacknewTreeSize < previousTreeSize for the origin’s latest checkpoint (CheckMonotonicityAsync)CriticalRejectAndAlert
CrossLogDivergencePrimary RootHash ≠ mirror RootHash at the same compared tree size (CheckCrossLogConsistencyAsync)WarningAlert
StaleCheckpointLatest checkpoint age > StaleCheckpointThreshold (DetectDivergenceAsync)WarningAlert
StaleTreeSizenewTreeSize == previousTreeSize and age > StaleTreeSizeThreshold (CheckMonotonicityAsync)InfoLog

Action column scope. RootHashMismatch, StaleCheckpoint, and StaleTreeSize are produced by DetectDivergenceAsync (which runs DetermineAction and so carries a real RecommendedAction); TreeSizeRollback is produced inside that same run via CheckMonotonicityAsync. CrossLogDivergence, however, is produced by the standalone CheckCrossLogConsistencyAsync, which returns a CrossLogConsistencyResult with no RecommendedAction field — its Alert entry above is the severity- implied action, not a value the method computes.

Severity enum (source ICheckpointDivergenceDetector.cs): None=0, Info=1, Warning=2, Error=3, Critical=4. A detection run is reported IsConsistent when every anomaly is below Error — i.e. Info/Warning anomalies do not flip a run to inconsistent.

Defined but not produced by the detector: the AnomalyType enum also declares InvalidSignature and ConsistencyProofFailure. The current CheckpointDivergenceDetector never raises these — they are placeholders for checkpoint signature/consistency-proof checks that are NOT IMPLEMENTED in the detector.

Alert Payloads

Alerts are emitted as NotifyEventEnvelope records by CheckpointDivergenceAlertPublisher. The envelope carries kind (the event type), tenant (defaults to system), ts, actor (attestor.divergence-detector), version (1.0), an attribute map, and a JSON payload. The JSON snippets below show the payloadbody.

The kind/eventType is mapped from the AnomalyType by GetEventKind: RootHashMismatch → rekor.checkpoint.divergence, TreeSizeRollback → rekor.checkpoint.rollback, StaleTreeSize → rekor.checkpoint.stale_size, CrossLogDivergence → rekor.checkpoint.cross_log_divergence, InvalidSignature → rekor.checkpoint.invalid_signature, StaleCheckpoint → rekor.checkpoint.stale, ConsistencyProofFailure → rekor.checkpoint.consistency_failure, fallback rekor.checkpoint.anomaly.

PublishDivergenceAlertAsync builds a single, uniform payload (BuildAlertPayload) for every single-anomaly event — including root-hash mismatch and rollback. expectedRootHash/actualRootHash are populated from the anomaly’s ExpectedValue/ActualValue, which are uppercase hex (Convert.ToHexString), not sha256:-prefixed, and for a rollback they hold ">= <prevSize>" / <newSize> rather than hashes. backend is set to the checkpoint origin (there is no separate backend identifier in the payload), and there is no recommendedAction field.

Root Hash Mismatch / Rollback Alert (uniform BuildAlertPayload)

{
  "eventType": "rekor.checkpoint.divergence",
  "severity": "critical",
  "origin": "rekor.sigstore.dev",
  "treeSize": 12345678,
  "expectedRootHash": "ABC123...",
  "actualRootHash": "DEF456...",
  "detectedAt": "2026-01-15T12:34:56.0000000+00:00",
  "backend": "rekor.sigstore.dev",
  "description": "Root hash mismatch at tree size 12345678",
  "anomalyType": "RootHashMismatch",
  "checkpointId": "00000000-0000-0000-0000-000000000000",
  "referenceCheckpointId": "00000000-0000-0000-0000-000000000000"
}

A rollback uses the same shape with eventType: "rekor.checkpoint.rollback", anomalyType: "TreeSizeRollback", expectedRootHash: ">= 12345678", actualRootHash: "12345600", and referenceCheckpointId: null.

Envelope attributes for PublishDivergenceAlertAsync are severity (lowercased), anomaly_type, and checkpoint_id.

Cross-Log Divergence Alert (PublishCrossLogDivergenceAlertAsync)

This path emits its own payload (no root-hash fields):

{
  "eventType": "rekor.checkpoint.cross_log_divergence",
  "severity": "warning",
  "primaryOrigin": "rekor.sigstore.dev",
  "mirrorOrigin": "rekor.mirror.example.com",
  "treeSize": 12345678,
  "checkedAt": "2026-01-15T12:34:56.0000000+00:00",
  "description": "Cross-log divergence detected between primary and mirror Rekor logs."
}

Envelope attributes for the cross-log alert are severity (warning), primary_origin, and mirror_origin.

Metrics

The detector instruments the meter StellaOps.Attestor.Divergence(v1.0.0). Instrument names use a dot between the namespace and the metric (attestor.<metric>); the exact Prometheus name depends on the OTel exporter’s dot/underscore normalization. The tag sets below are exactly what the source emits.

# Counter: total checkpoint mismatches (tags: origin, backend="primary" — backend is hardcoded)
attestor.rekor_checkpoint_mismatch_total{origin="rekor.sigstore.dev",backend="primary"} 0

# Counter: rollback attempts detected (tag: origin)
attestor.rekor_checkpoint_rollback_detected_total{origin="rekor.sigstore.dev"} 0

# Counter: cross-log divergences detected (tags: primary, mirror)
attestor.rekor_cross_log_divergence_total{primary="rekor.sigstore.dev",mirror="rekor.mirror.example.com"} 0

# Counter: total anomalies detected, incremented by anomaly count per run (tag: origin only — no type/severity tags)
attestor.rekor_anomalies_detected_total{origin="rekor.sigstore.dev"} 0

NOT IMPLEMENTED — attestor_rekor_checkpoint_age_seconds gauge. The divergence detector does not export a checkpoint-age gauge. Checkpoint age is computed in-memory by GetLogHealthAsync but is not instrumented by this meter. A related metric is defined in source, but on a different meter and as a histogram: RekorSyncBackgroundService (meter StellaOps.Attestor.RekorSync) creates attestor.rekor_sync_checkpoint_age_seconds (unit s) along with attestor.rekor_sync_checkpoints_fetched, attestor.rekor_sync_tiles_fetched, and the observable gauge attestor.rekor_sync_tiles_cached. Caveat: because RekorSyncBackgroundService is not registered as a hosted service (see Implementation status), this meter does not emit in the running Attestor host today either — the histogram only records under the sync service’s unit tests. There is no live “seconds since last checkpoint” signal at present; this is roadmap work, not an alertable metric you can wire up now.

Incident Response Procedures

Level 1: Root Hash Mismatch (CRITICAL)

Symptoms:

Immediate Actions:

  1. Quarantine all affected proofs - Do not rely on any inclusion proofs from the affected log until resolved
  2. Suspend automated verifications - Halt any automated systems that depend on the log
  3. Preserve evidence - Capture both checkpoints (expected and actual) with full metadata
  4. Alert security team - This is a potential compromise indicator

Investigation Steps:

  1. Verify the mismatch isn’t local storage corruption. Inspect the two stored checkpoints directly in the Attestor checkpoint store (the records compared are StoredCheckpoint rows keyed by Origin + TreeSize; compare their RootHash, RawCheckpoint, and Signature). NOTE: there is currently no dedicated stella CLI verb for this — checkpoint inspection is a store-level / DB query, not a shipped operator command. NOTE (status): IRekorCheckpointStore is not DI-registered in the running host (see Implementation status), so there is no live checkpoint table to query today; this step describes the designed workflow once the store is wired.
  2. Cross-check with independent sources (other clients, mirrors)
  3. Check if Sigstore has published any incident reports
  4. Review network logs for MITM indicators

Resolution:

Level 2: Tree Size Rollback (CRITICAL)

Symptoms:

Immediate Actions:

  1. Reject the checkpoint - Do not accept or store it
  2. Log full details for forensic analysis
  3. Check network path - Could indicate MITM or DNS hijacking

Investigation Steps:

  1. Verify current log state directly:
    curl -s https://rekor.sigstore.dev/api/v1/log | jq .treeSize
  2. Compare with stored latest tree size
  3. Check DNS resolution and TLS certificate chain

Resolution:

Level 3: Cross-Log Divergence (WARNING)

Symptoms:

Immediate Actions:

  1. Do not panic - Mirrors may have legitimate lag
  2. Check mirror sync status - May be catching up

Investigation Steps:

  1. Compare the latest stored tree sizes for the two origins in the checkpoint store. CheckCrossLogConsistencyAsync compares at the smaller of the two tree sizes (Math.Min); if a checkpoint at that size is missing for either origin it returns IsConsistent = true (cannot-verify is treated as consistent), so a transient mirror gap will not raise an anomaly. NOTE: no stella CLI verb lists checkpoints today — this is a store/DB query.
  2. If same tree size with different roots: Escalate to CRITICAL
  3. If different tree sizes: Allow time for sync
  4. If persistent: Investigate mirror operator

Resolution:

Level 4: Stale Checkpoint (WARNING)

Symptoms:

Immediate Actions:

  1. Check log service status
  2. Verify network connectivity to log

Investigation Steps:

  1. Check Sigstore status page
  2. Test direct API access:
    curl -I https://rekor.sigstore.dev/api/v1/log
  3. Review recent checkpoint fetch attempts

Resolution:

Configuration

Binding caveat (verify against src/). The detector and publisher read strongly-typed options records — DivergenceDetectorOptions and DivergenceAlertOptions (defined in CheckpointDivergenceDetector.cs / CheckpointDivergenceAlertPublisher.cs). Because the detector/publisher are not yet registered in DI, there is no canonical configuration-section prefix bound in source. The YAML below is illustrative of the option properties and their source defaults; the section names (attestor:divergenceDetection, attestor:alerts) are a suggested layout, not a key path the running service reads today. Fields are the actual option property names (camelCased) and their default values.

Detector Options — DivergenceDetectorOptions

# Illustrative section name; properties/defaults match DivergenceDetectorOptions
divergenceDetection:
  # Latest checkpoint older than this raises a StaleCheckpoint (Warning) anomaly. Default 15m.
  staleCheckpointThreshold: 15m

  # Tree size unchanged longer than this raises a StaleTreeSize (Info) anomaly. Default 1h.
  staleTreeSizeThreshold: 1h

  # Log health thresholds used by GetLogHealthAsync.
  degradedCheckpointAgeThreshold: 30m
  unhealthyCheckpointAgeThreshold: 2h

  # Enable cross-log consistency checks. Default true.
  enableCrossLogChecks: true

  # Mirror origins to check against primary. Default: empty list.
  mirrorOrigins:
    - rekor.mirror.example.com
    - rekor.mirror2.example.com

There is no enabled flag on DivergenceDetectorOptions — the original “enable checkpoint monitoring” toggle does not exist in source. Enablement is governed by whether the detector is wired in at all.

Alert Options — DivergenceAlertOptions

# Illustrative section name; properties/defaults match DivergenceAlertOptions
alerts:
  # Master switch for alert publishing (property: EnableAlerts). Default true.
  enableAlerts: true

  # Default tenant on the NotifyEventEnvelope. Default "system".
  defaultTenant: system

  # Severity thresholds (Critical always alerts regardless of these flags).
  alertOnHighSeverity: true   # alert on Error severity. Default true.
  alertOnWarning: true        # Default true.
  alertOnInfo: false          # Default false (not recommended for production).

  # Notify stream name (property: AlertStream). Default "attestor.alerts".
  alertStream: attestor.alerts

ShouldAlert always alerts on Critical; Error is gated by alertOnHighSeverity, Warning by alertOnWarning, Info by alertOnInfo. The property is enableAlerts/alertStream, not enabled/stream as previously documented.

alertStream is currently inert. The property exists on DivergenceAlertOptions but the publisher never reads it: PublishDivergenceAlertAsync/PublishCrossLogDivergenceAlertAsync build a NotifyEventEnvelope (which has no stream field) and hand it to INotifyEventPublisher.PublishAsync — the stream/destination is the publisher implementation’s concern, not this option. Setting alertStream has no effect on where alerts land today.

Runbook Checklist

The checklist below describes the designed daily/weekly operations once the checkpoint-sync and divergence pipeline is wired into the host. Today none of these signals are emitted by the running Attestor service (the sync service and detector are unregistered library code — see Implementation status), so the checklist is roadmap guidance, not a current operator procedure.

Daily Operations

Weekly Review

Post-Incident

See Also