Rekor Checkpoint Sync Configuration and Operations

Audience: Operators configuring offline/air-gapped attestation verification in Stella Ops.

This guide covers the configuration and operational procedures for the Rekor periodic checkpoint synchronization service, which maintains a local mirror of transparency-log checkpoints and tiles. For the submission/retry side of the Attestor’s Rekor integration, see the companion Rekor Submission & Retry Policy.

Implementation status (verified against source 2026-05-30). The sync engine (RekorSyncBackgroundService), checkpoint store (PostgresRekorCheckpointStore), and tile cache (FileSystemRekorTileCache) are implemented in src/Attestor/StellaOps.Attestor/StellaOps.Attestor.Core/Rekor/ (plus …/StellaOps.Attestor.Storage/Rekor/). However, as of this writing the background service is not yet registered as a hosted service and its options (RekorSyncOptions, PostgresCheckpointStoreOptions, FileSystemTileCacheOptions) are not yet bound to any configuration section in AttestorWebServiceComposition. The configuration tree shown below documents the option records as they exist in code; the binding/wiring is roadmap work. Likewise, the only stella attestor … CLI surface that currently exists is the external transparency mirror group (stella attestor transparency …, see CLI reference) — the checkpoint-store / sync / tiles / export / import commands referenced in older drafts of this guide are NOT IMPLEMENTED. Sections that describe unimplemented surfaces are flagged inline.

Overview

The Rekor sync service maintains a local mirror of Rekor transparency log checkpoints and tiles. This enables:

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      RekorSyncBackgroundService                  │
│  ┌──────────────┐     ┌──────────────┐     ┌──────────────┐     │
│  │  Checkpoint  │     │   Signature  │     │    Tile      │     │
│  │   Fetcher    │────▶│   Verifier   │────▶│   Syncer     │     │
│  └──────────────┘     └──────────────┘     └──────────────┘     │
└─────────────────────────────────────────────────────────────────┘
          │                     │                     │
          ▼                     ▼                     ▼
   ┌──────────────┐      ┌──────────────┐     ┌──────────────┐
   │  HTTP Tile   │      │  Checkpoint  │     │    Tile      │
   │   Client     │      │    Store     │     │   Cache      │
   └──────────────┘      │ (PostgreSQL) │     │(File System) │
          │              └──────────────┘     └──────────────┘
          ▼
   ┌──────────────┐
   │    Rekor     │
   │   Server     │
   └──────────────┘

Configuration

Draft / not yet wired. The option records below exist in code, but no Configure<…>(GetSection(…)) binding currently exists for them. Section names and binding semantics are therefore provisional — the keys are derived from the C# property names (.NET configuration binds case-insensitively), but the root section path is not fixed by code yet. Treat this as the target shape, not a working configuration.

Basic Configuration (RekorSyncOptions)

Source: RekorSyncOptions in src/Attestor/StellaOps.Attestor/StellaOps.Attestor.Core/Rekor/RekorSyncBackgroundService.cs.

# Provisional section path — not yet bound in AttestorWebServiceComposition.
rekorSync:
  # Enable or disable sync service. Default: true.
  enabled: true

  # Interval between sync cycles (TimeSpan). Default: 00:05:00 (5 minutes).
  syncInterval: "00:05:00"

  # Delay before first sync after startup (TimeSpan). Default: 00:00:30 (30 seconds).
  initialDelay: "00:00:30"

  # Sync tiles in addition to checkpoints, for full offline support. Default: true.
  enableTileSync: true

  # Maximum number of tiles to fetch per sync cycle. Default: 100.
  maxTilesPerSync: 100

  # Rekor backends to sync (RekorBackend). Defaults to a single sigstore-prod entry.
  backends:
    - name: sigstore-prod                                   # required
      url: "https://rekor.sigstore.dev"                     # required (absolute URI)
      tileBaseUrl: "https://rekor.sigstore.dev/api/v1/log/tiles"
      # version: Auto | V2   (default Auto, which selects v2 tiles)
      # logId: "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"
      # publicKey: <byte[]>  (PEM/SPKI bytes; if omitted, checkpoint signatures are NOT verified)
      # proofTimeout: "00:00:15"   (TimeSpan; default 15s — inclusion-proof fetch timeout)
      # pollInterval: "00:00:00.250" (TimeSpan; default 250ms — proof poll cadence)
      # maxAttempts: 60            (int; default 60 — proof poll attempts)

Field-name note. A backend is identified by name and url — there are no id, origin, baseUrl, or publicKeyPath fields on RekorBackend. The full property set on RekorBackend (source: RekorBackend.cs) is: Name, Url, Version, TileBaseUrl, LogId, ProofTimeout (default 15s), PollInterval (default 250ms), MaxAttempts (default 60), and PublicKey. The log origin is read from each fetched checkpoint (RekorTileCheckpoint.Origin), not from backend configuration. The trusted log key is supplied as in-memory bytes via PublicKey (no file-path option exists in code today). ProofTimeout/PollInterval/MaxAttempts govern inclusion-proof fetching and are not consumed by the checkpoint/tile sync loop.

Checkpoint Store Configuration (PostgresCheckpointStoreOptions)

Source: PostgresCheckpointStoreOptions in src/Attestor/StellaOps.Attestor/StellaOps.Attestor.Storage/Rekor/PostgresRekorCheckpointStore.cs.

checkpointStore:
  connectionString: "Host=localhost;Database=stella;Username=stella;Password=secret"  # required
  schema: attestor          # default: "attestor"
  autoInitializeSchema: true # default: true

Checkpoints are stored in attestor.rekor_checkpoints with a unique (origin, tree_size) constraint; re-fetching the same (origin, tree_size) updates fetched_at / verified / verified_at via ON CONFLICT … DO UPDATE.

Schema-creation correction (verified against source 2026-05-30). A source comment in PostgresRekorCheckpointStore.cs claims the table is “handled by the central migration runner (AttestorMigrationModulePlugin)”, but that is not accurate: AttestorMigrationModulePlugin (src/Platform/__Libraries/StellaOps.Platform.Database/MigrationModulePlugins.cs) manages the proofchainschema, and the AddStartupMigrations(schemaName: "attestor", …) runner only applies the nine embedded SQL files under StellaOps.Attestor.Persistence/Migrations/ (the _archived/** subtree is excluded by the .csproj glob). Those files cover the verdict ledger, the consolidated 001_initial_schema proofchain/queue bootstrap, the predicate-type registry, the artifact-canonical-record materialized view, the deprecated-audit-log drop, runtime entries/watchlist, and the local-transparency-entries/external-transparency-sync tables — and span both the proofchain and attestor schemas — but none of them creates rekor_checkpoints. No migration anywhere in src/ creates this table. The only code path that creates attestor.rekor_checkpoints is PostgresRekorCheckpointStore.InitializeSchemaAsync(), which issues an idempotent CREATE TABLE IF NOT EXISTS (plus the two indexes shown below) when AutoInitializeSchema is true. Because the store is not yet registered/called by a running service (see the status note at the top of this guide), even that path is not exercised today — the table must be created out of band until the wiring lands.

Tile Cache Configuration (FileSystemTileCacheOptions)

Source: FileSystemTileCacheOptions in src/Attestor/StellaOps.Attestor/StellaOps.Attestor.Core/Rekor/FileSystemRekorTileCache.cs.

tileCache:
  # Base directory for tile storage.
  # Default: <LocalApplicationData>/StellaOps/RekorTiles
  basePath: /var/lib/stella/attestor/tiles

  # Maximum cache size in bytes (0 = unlimited). Default: 0.
  # NOTE: persisted as an option but NOT currently enforced by the file-system cache.
  maxCacheSizeBytes: 10737418240  # 10 GB

  # Auto-prune duration (TimeSpan, optional/nullable). Default: unset.
  # NOTE: the cache exposes PruneAsync(...) but does not auto-schedule it from this value today.
  autoPruneAfter: "30.00:00:00"  # 30 days

Tiles are laid out on disk as {basePath}/{sanitized-origin}/{level}/{index}.tile with a sibling {index}.meta.json. Standard tile width is 256 hashes (32 bytes each); tiles narrower than 256 hashes are recorded as partial.

Operational Behaviour

The sync service runs as a hosted BackgroundService (once registered). Its loop, implemented in RekorSyncBackgroundService.ExecuteAsync, is:

  1. If Enabled is false, log and return without doing anything.
  2. Wait InitialDelay, then enter the sync loop.
  3. Each cycle iterates every configured backend (SyncAllBackendsAsync); a failure in one backend is logged as a warning and does not abort the cycle.
  4. For each backend (SyncBackendAsync):
    • Fetch the latest checkpoint via IRekorTileClient.GetCheckpointAsync. If none is available, log a warning and skip.
    • Verify the checkpoint signature via IRekorCheckpointVerifier.VerifyCheckpointAsync. On failure, log an error and do not store the checkpoint.
    • If an IRekorConsistencySyncCoordinator is wired, run an online consistency cycle (G9-REK-02). If the verdict is not healthy (suspected fork/truncation), refuse to store so the consumer-facing “last good” pin does not advance.
    • Store the checkpoint via IRekorCheckpointStore.StoreCheckpointAsync (idempotent on (origin, tree_size)).
    • If EnableTileSync is true, perform incremental tile sync (SyncTilesAsync): compute the new-entry range from the previously stored checkpoint’s tree size, ask the cache for missing tiles, and fetch up to MaxTilesPerSync of them per cycle.
  5. Sleep SyncInterval and repeat until cancellation.

There is no built-in “full” or “initial” sync mode and no per-cycle forced-resync switch — the loop always performs an incremental tile fetch bounded by MaxTilesPerSync, catching up over successive cycles. Backfill therefore happens implicitly across multiple cycles rather than via a one-shot --full flag.

CLI commands — NOT IMPLEMENTED

The following commands appeared in earlier drafts of this guide but do not exist in the stella CLI today. There is no checkpoint-store, backend, sync, sync-status, checkpoints, tiles, export, import, or config validate subcommand under stella attestor for the checkpoint/tile sync engine:

stella attestor checkpoint-store init        # NOT IMPLEMENTED
stella attestor backend add …                # NOT IMPLEMENTED
stella attestor sync …                        # NOT IMPLEMENTED (see transparency sync below)
stella attestor sync-status                   # NOT IMPLEMENTED
stella attestor checkpoints list|prune|verify # NOT IMPLEMENTED
stella attestor tiles stats|prune|clear       # NOT IMPLEMENTED
stella attestor export / import               # NOT IMPLEMENTED (see transparency exchange below)
stella attestor config validate               # NOT IMPLEMENTED

Operationally, the checkpoint/tile sync engine is driven by the background service and its configuration, not by interactive CLI commands. Checkpoint history is inspected directly in PostgreSQL (attestor.rekor_checkpoints); tile-cache contents are inspected on disk under the configured basePath.

One nuance on export/import: a rekor checkpoint export|import|status command group is implemented in CheckpointCommands.cs but is not registered in CommandFactory.cs, so it is unreachable from the shipped CLI (see Export checkpoints for air-gap under Maintenance Tasks). It is therefore effectively unavailable, even though the C# code exists.

CLI reference: stella attestor transparency

The CLI surface that does exist under stella attestor is the external transparency mirror group (source: src/Cli/StellaOps.Cli/Commands/AttestorTransparencyCommandGroup.cs). This governs mirroring local attestations to external transparency logs — it is related to, but distinct from, the local checkpoint/tile sync described above.

stella attestor transparency targets list [--enabled <bool>] [--json]
stella attestor transparency targets show <target> [--json]
stella attestor transparency targets upsert --name … --required-crypto-profile … --operator-label … [--kind …] [--endpoint-url …] [--trust-root-ref …] [--auth-ref …] [--metadata-json …] [--disabled] [--json]
stella attestor transparency targets enable|disable <target> [--json]

stella attestor transparency sync run --target <name|id> … [--live] [--confirm-egress-window] [--dry-run] [--max-entries N] [--max-runtime-seconds N] [--stop-on-first-hard-error] [--operator-label …] [--json]
stella attestor transparency sync report [--target …] [--batch-id <guid>] [--operator-label …] [--limit N] [--json]

stella attestor transparency receipts list [filters…] [--json]
stella attestor transparency receipts show <uuid> [--refresh] [--json]

stella attestor transparency exchange schedule create …
stella attestor transparency exchange relay-import create …
stella attestor transparency exchange import validate|apply …
stella attestor transparency exchange status …

Notes:

Metrics

The sync service emits four OpenTelemetry instruments from the StellaOps.Attestor.RekorSync meter (source: RekorSyncBackgroundService.cs). All instrument names use dots; the names below are the raw OTel instrument names. The Prometheus exporter conventionally replaces dots with underscores and appends _total to monotonic counters, so the exported series names are shown in the comments. All series are tagged with origin (the log origin from the checkpoint), not backend.

# Counter (long): checkpoints fetched from remote.
#   Prometheus: attestor_rekor_sync_checkpoints_fetched_total{origin="rekor.sigstore.dev"}
attestor.rekor_sync_checkpoints_fetched

# Counter (long): tiles fetched from remote.
#   Prometheus: attestor_rekor_sync_tiles_fetched_total{origin="rekor.sigstore.dev"}
attestor.rekor_sync_tiles_fetched

# Histogram (double, unit "s"): age of the latest synced checkpoint at sync time.
#   Prometheus: attestor_rekor_sync_checkpoint_age_seconds_bucket/_sum/_count{origin="rekor.sigstore.dev"}
attestor.rekor_sync_checkpoint_age_seconds

# Observable gauge (long): number of tiles currently cached (last observed value).
#   Prometheus: attestor_rekor_sync_tiles_cached{origin="rekor.sigstore.dev"}
attestor.rekor_sync_tiles_cached

Removed/never-existed metrics. Earlier drafts listed attestor_rekor_sync_checkpoints_stored_total, attestor_rekor_sync_tiles_cached_total, attestor_rekor_sync_last_success_seconds, and attestor_rekor_sync_errors_total. None of these instruments exist in code. There is no “checkpoints stored” counter (storage is logged, not metered), no “time since last successful sync” gauge, and no sync-error counter on the sync meter.

Divergence-detection metrics (separate meter)

Checkpoint divergence/anomaly detection (CheckpointDivergenceDetector) emits from a different meter, StellaOps.Attestor.Divergence. These are already-_total counters in code:

attestor.rekor_checkpoint_mismatch_total{origin,backend}
attestor.rekor_checkpoint_rollback_detected_total{origin}
attestor.rekor_cross_log_divergence_total{primary,mirror}
attestor.rekor_anomalies_detected_total{origin}

Alerting Recommendations

Because there is no last_success_seconds gauge or errors_total counter, build freshness alerting on the checkpoint-age histogram and lack of fetch progress instead.

groups:
  - name: attestor-rekor-sync
    rules:
      # Latest synced checkpoint is aging — derive from the age histogram.
      - alert: RekorCheckpointAgeHigh
        expr: histogram_quantile(0.95, rate(attestor_rekor_sync_checkpoint_age_seconds_bucket[15m])) > 900
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Rekor checkpoint age is high
          description: "p95 checkpoint age > 900s for {{ $labels.origin }}"

      # No checkpoints fetched in the last 15m — sync may be stalled.
      - alert: RekorSyncStalled
        expr: increase(attestor_rekor_sync_checkpoints_fetched_total[15m]) == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: Rekor sync appears stalled
          description: "No checkpoints fetched for {{ $labels.origin }} in 15m"

      # Integrity anomalies — wire to the divergence meter.
      - alert: RekorCheckpointDivergence
        expr: increase(attestor_rekor_checkpoint_mismatch_total[5m]) > 0
              or increase(attestor_rekor_checkpoint_rollback_detected_total[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: Rekor checkpoint divergence/rollback detected
          description: "Integrity anomaly for {{ $labels.origin }} — see checkpoint-divergence-runbook.md"

Maintenance Tasks

No CLI/scheduler surface today. Pruning is exposed only as code-level APIs; there are no stella attestor checkpoints prune, tiles prune, checkpoints verify, or export commands, and the background service does not auto-schedule pruning.

Prune old checkpoints — code API: IRekorCheckpointStore.PruneOldCheckpointsAsync(olderThan, keepLatestPerOrigin = true). Deletes checkpoint rows older than the cutoff; with keepLatestPerOrigin = true (default) it always retains the newest checkpoint per origin. To prune manually in the absence of a CLI, delete from attestor.rekor_checkpoints by fetched_at.

Prune old tiles — code API: IRekorTileCache.PruneAsync(origin, olderThan). Deletes .tile files (and their .meta.json siblings) whose file creation time is older than the cutoff, scoped to one origin or all origins. The FileSystemTileCacheOptions.AutoPruneAfter value is stored but not auto-applied by the cache; pruning must be invoked explicitly.

Verify checkpoint store integrity — there is no dedicated verify command. Signature verification happens at sync time (IRekorCheckpointVerifier); divergence and rollback detection is handled by CheckpointDivergenceDetector (see the divergence runbook).

Export checkpoints for air-gap — there is no reachable CLI for native checkpoint bundle export/import. A rekor checkpoint export|import|status command group does exist in code (src/Cli/StellaOps.Cli/Commands/CheckpointCommands.cs: BuildCheckpointCommand, with export --instance --output [--include-tiles --tile-count], import --input [--verify-signature --force], and status [--output table|json]), but BuildCheckpointCommand is never registered in CommandFactory.cs, so it is unreachable from the stella CLI today — treat it as roadmap/dead code, not a usable surface. For external transparency mirroring, use the stella attestor transparency exchange commands (see the CLI reference); for general offline distribution see the Offline Kit documentation under docs/modules/airgap/ and docs/modules/scanner/operations/offline-kit-assembly.md.

Troubleshooting

Sync Not Running

  1. Confirm the service is actually wired. The most common reason for “no sync” today is that RekorSyncBackgroundService is not yet registered as a hosted service and RekorSyncOptions is not yet bound to configuration (see the status note at the top of this guide). Until that wiring lands, the loop never starts regardless of configuration.

  2. Check service logs (the loop logs “Rekor sync service started …” on start, or “Rekor sync service is disabled” when Enabled is false):

    journalctl -u stella-attestor -f

  3. Check database connectivity directly (there is no checkpoint-store test command):

    psql -h localhost -U stella -d stella -c "SELECT count(*) FROM attestor.rekor_checkpoints"

Signature Verification Failing

When IRekorCheckpointVerifier.VerifyCheckpointAsync returns invalid, the service logs Checkpoint signature verification failed for {Origin} at error level and skips storing the checkpoint.

  1. Verify the configured trusted key. The log key is supplied as RekorBackend.PublicKey bytes (no verify-key command exists). If PublicKey is omitted, checkpoint signatures are not verified at all.

  2. Check for key rotation:

    • Monitor Sigstore announcements.
    • Update PublicKey (and LogId if applicable) if the upstream log key rotated.

  3. Compare with a direct fetch (only when not in offline/sealed mode; web access is user-initiated per the offline-first policy):

    curl -s https://rekor.sigstore.dev/api/v1/log | jq

Tile Cache Issues

  1. Check disk space:

    df -h /var/lib/stella/attestor/tiles

  2. Verify permissions:

    ls -la /var/lib/stella/attestor/tiles

  3. Clear and resync. There is no stella attestor tiles clear or stella attestor sync --full-tiles command (see CLI commands — NOT IMPLEMENTED). To clear the cache, delete the on-disk tiles for an origin (or the whole tree) under the configured basePath; the next background sync cycle re-fetches the missing tiles incrementally (bounded by maxTilesPerSync) once the sync service is wired:

    # Remove cached tiles for one origin (sanitized-origin subdir under basePath)…
rm -rf /var/lib/stella/attestor/tiles/<sanitized-origin>/
# …or clear all origins:
rm -rf /var/lib/stella/attestor/tiles/*

Database Issues

  1. Check PostgreSQL connectivity:

    psql -h localhost -U stella -d stella -c "SELECT 1"

  2. Verify schema/columns exist (the table schema is: checkpoint_id UUID PK, origin TEXT, tree_size BIGINT, root_hash BYTEA, raw_checkpoint TEXT, signature BYTEA, fetched_at TIMESTAMPTZ, verified BOOLEAN, verified_at TIMESTAMPTZ, with UNIQUE(origin, tree_size)):

    SELECT origin, tree_size, verified, fetched_at
FROM attestor.rekor_checkpoints
ORDER BY fetched_at DESC
LIMIT 1;

  3. Schema creation is not owned by a migration. Despite a misleading source comment, AttestorMigrationModulePlugin manages the proofchain schema and the attestor-schema migration runner does not create rekor_checkpoints (see the schema-creation correction under Checkpoint Store Configuration above). The only code that creates the table is PostgresRekorCheckpointStore.InitializeSchemaAsync (idempotent CREATE TABLE IF NOT EXISTS when AutoInitializeSchema is true), and that store is not yet wired into a running service. There is no checkpoint-store init --force command. Until the background service/store are registered, create the table out of band with the DDL listed in step 2 above.

Air-Gap Operations

No reachable CLI for checkpoint/tile bundle export/import. The stella attestor … commands shown in older drafts (stella attestor sync --full-tiles, stella attestor export, stella attestor import, stella attestor sync-status) do not exist under the attestor namespace. A separate rekor checkpoint export|import|status group is coded in CheckpointCommands.cs but is not registered in CommandFactory.cs, so it cannot be invoked from the shipped CLI (see Maintenance Tasks). For now, treat checkpoint/tile bundle export/import as unavailable.

For air-gapped transparency operations, use the implemented external-transparency controlled-media exchange flow instead (source: AttestorTransparencyCommandGroup):

  1. On the connected/sealed site, create a schedule package:

    stella attestor transparency exchange schedule create \
  --target <name|id> --out ./pkg --media-id <id> --operator-label <label>

  2. On a connected relay, build the import package from the schedule package:

    stella attestor transparency exchange relay-import create \
  --schedule ./pkg --out ./import-pkg --relay-id <id> --operator-label <label>

  3. Validate and apply the returned import package on the sealed site:

    stella attestor transparency exchange import validate --package ./import-pkg
stella attestor transparency exchange import apply    --package ./import-pkg --operator-label <label>

  4. Inspect exchange/sync status:

    stella attestor transparency exchange status --schedule-id <guid>
stella attestor transparency sync report --target <name|id>

For populating local checkpoint/tile state in an air-gapped install, the engine relies on the configured tile cache (basePath) being pre-seeded out of band, since no first-class export/import tool exists yet. See the Offline Kit documentation under docs/modules/airgap/ and docs/modules/scanner/operations/offline-kit-assembly.md for distribution mechanics.

See Also