Rekor Transparency Log Submission & Retry Policy

Last Updated: 2026-05-30 Owner: Attestor Team Audience: Operators running the Stella Ops Attestor Sprint: SPRINT_3000_0001_0002_rekor_retry_queue_metrics

Current architecture note (2026-07-17). New Attestor submissions do not enqueue attestor.rekor_submission_queue. Attestor commits its authoritative local transparency entry first; optional connected/manual publication uses ITransparencySyncRunner and durable external mirror receipts. The queue, metrics, and RekorRetryWorker sections below document legacy drain compatibility for rows created by older releases, not the current submission path. Do not use this page to design a new queue producer.

Overview

This document retains the legacy remote-Rekor queue lifecycle and operator recovery details. The compatibility transport remains implemented for draining pre-existing rows, but current submission durability belongs to the local-first transparency and external-sync architecture; the tier/budget policy described in earlier drafts of this page is not implemented and is tracked as roadmap (see Roadmap: tiers & budgets).

Reconciliation note (2026-05-30). A prior version of this page documented “submission tiers”, per-tenant hourly/burst/daily budget limits, edge-bundle escalation triggers, attestor.rekor_quotas / attestor.rekor_offline_queue tables, and a family of attestor_rekor_budget_* / attestor_rekor_submissions_* metrics. None of those exist in the codebase. The real implementation is a durable submission retry queue (attestor.rekor_submission_queue, IRekorSubmissionQueue / PostgresRekorSubmissionQueue) plus a verification job. This page has been corrected to match the source. The aspirational tier/budget model is retained, clearly marked, under Roadmap.

Ground-truth source symbols:

Durable Submission Queue

Rekor submissions are persisted to a durable PostgreSQL-backed queue so that a transient Rekor outage, network failure, or process restart never loses an attestation. Submission happens asynchronously: a caller enqueues a DSSE payload, and RekorRetryWorker drains the queue in the background.

Queue lifecycle

Each queue item moves through the RekorSubmissionStatus states:

pending -> submitting -> submitted
                       \-> retrying -> ... -> dead_letter

Backends

Each item targets a Rekor backend — either primary or mirror (IRekorSubmissionQueue.EnqueueAsync(..., string backend, ...)). Backend resolution and failover (including circuit-breaker behavior) is handled by RekorBackendResolver / ResilientRekorClient. There is no per-tenant “submission tier” concept in the queue; tenant_id is stored only for scoping and lookups.

Retry & backoff

Retry behavior comes entirely from RekorQueueOptions. There is no rate budget, quota, or 429-style admission control on Rekor submissions; the only limits are retry count, backoff, and batch size:

Option (RekorQueueOptions)DefaultMeaning
EnabledtrueEnable the durable retry queue/worker.
MaxAttempts5Attempts before dead-lettering. Stored per-item as max_attempts.
InitialDelayMs1000Base delay for the first retry.
MaxDelayMs60000Delay ceiling (1 minute).
BackoffMultiplier2.0Exponential multiplier: InitialDelayMs * multiplier^attempt, capped at MaxDelayMs.
BatchSize10Items dequeued per worker pass.
PollIntervalMs5000Worker poll interval.
DeadLetterRetentionDays30Dead-letter items older than this are purged (0 = indefinite).

RekorQueueOptions binds from the configuration section Attestor:rekorQueue (see AttestorWebServiceComposition.cs).

Dead-letter management

PostgresRekorSubmissionQueue exposes operational entry points for the dead-letter set: GetDeadLetterItemsAsync, GetDeadLetterCountAsync, RequeueDeadLetterAsync (resets an item back to pending, attempt_count = 0), and PurgeDeadLetterAsync(retentionDays). Successfully-submitted rows can be trimmed with PurgeSubmittedAsync(olderThan).

Queue Schema

The actual table is attestor.rekor_submission_queue. It is the only Rekor queue table; there is no rekor_quotas table and no separate rekor_offline_queue table.

Active migration (deployed authority): 001_initial_schema.sql under src/Attestor/__Libraries/StellaOps.Attestor.Persistence/Migrations/, applied on startup by AddStartupMigrations(schemaName: "attestor", moduleName: "Attestor.Persistence", ...) in AttestorWebServiceComposition.cs. That file is a 1.0.0 compaction that consolidated the original 20251216_001_create_rekor_submission_queue.sql (Sprint SPRINT_3000_0001_0002). The original now lives only under StellaOps.Attestor.Infrastructure/Migrations/_archived/pre_1.0/ and is not the migration that runs — treat 001_initial_schema.sql as authoritative.

CREATE SCHEMA IF NOT EXISTS attestor;

CREATE TABLE IF NOT EXISTS attestor.rekor_submission_queue (
    id              UUID PRIMARY KEY,
    tenant_id       TEXT NOT NULL,
    bundle_sha256   TEXT NOT NULL,
    dsse_payload    BYTEA NOT NULL,
    backend         TEXT NOT NULL DEFAULT 'primary',

    -- Status lifecycle: pending -> submitting -> submitted | retrying -> dead_letter
    status          TEXT NOT NULL DEFAULT 'pending'
                    CHECK (status IN ('pending', 'submitting', 'retrying', 'submitted', 'dead_letter')),

    attempt_count   INTEGER NOT NULL DEFAULT 0,
    max_attempts    INTEGER NOT NULL DEFAULT 5,
    next_retry_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    -- Populated on success
    rekor_uuid      TEXT,
    rekor_index     BIGINT,

    -- Populated on failure
    last_error      TEXT
);

Indexes (from the migration): a partial dequeue index on (status, next_retry_at) WHERE status IN ('pending','retrying'), a tenant index on (tenant_id), a bundle-dedupe index on (tenant_id, bundle_sha256), a dead-letter index on (status, updated_at) WHERE status = 'dead_letter', and a completed-cleanup index on (status, updated_at) WHERE status = 'submitted'.

The active 001_initial_schema.sql migration also installs a BEFORE UPDATE trigger (update_rekor_queue_updated_at) that calls proofchain.update_updated_at_column() to keep updated_at current on every row mutation. (Application code also sets updated_at explicitly, so the trigger is a backstop; the standalone archived DDL above does not include it.)

Monitoring

Metrics

All Attestor metrics are emitted on the meter StellaOps.Attestor (AttestorMetrics.MeterName). The Rekor queue metrics actually defined are:

Metric (instrument name)TypeDescriptionTags
attestor.rekor_queue_depthObservableGaugeCurrent queue depth (pending + retrying).—
attestor.rekor_submission_status_totalCounterSubmission status transitions.status, backend
attestor.rekor_retry_attempts_totalCounterRetry attempts.backend, attempt
attestor.rekor_queue_wait_secondsHistogramTime items spend waiting in the queue.—
attestor.rekor_dead_letter_totalCounterItems moved to dead letter.backend

A second set of inclusion/checkpoint instruments is declared on the same StellaOps.Attestor meter in AttestorMetrics: attestor.rekor_inclusion_verify_total, attestor.rekor_inclusion_verify_latency_seconds, attestor.rekor_checkpoint_verify_total, attestor.rekor_checkpoint_verify_latency_seconds, attestor.rekor_offline_verify_total, attestor.rekor_checkpoint_cache_hits, attestor.rekor_checkpoint_cache_misses.

Declared but not emitted (verify against source before relying on these). As of this revision these seven instruments are created in AttestorMetrics but have no .Add()/.Record() call site anywhere in src/ — they will register on the meter yet always read zero. They are not the metrics the periodic verification job produces; see the next subsection.

The periodic verification job (RekorVerificationJob) does not use AttestorMetrics. It emits its own counters/histograms via RekorVerificationMetrics on a separate meter StellaOps.Attestor.RekorVerification, with underscore-style names, e.g.: attestor_rekor_verification_runs_total, attestor_rekor_entries_verified_total, attestor_rekor_entries_failed_total, attestor_rekor_entries_skipped_total, attestor_rekor_time_skew_violations_total, attestor_rekor_signature_failures_total, attestor_rekor_inclusion_proof_failures_total, attestor_rekor_root_consistency_checks_total, attestor_rekor_root_inconsistencies_total, attestor_rekor_verification_run_failures_total, attestor_rekor_entry_verification_duration_seconds, attestor_rekor_batch_verification_duration_seconds, and attestor_rekor_verification_failure_rate. These are recorded via .Add()/.Record() inside RekorVerificationMetrics (so they emit when the job runs), but note the caveat in Periodic Verification: the RekorVerificationJob is not registered as a hosted service and the StellaOps.Attestor.RekorVerification meter is not added to the Attestor’s OpenTelemetry WithMetrics pipeline. To observe periodic-verification health you must first host the job and add this meter to your exporter; until then these instruments will not be scraped by the default pipeline.

Not implemented. Earlier drafts referenced attestor_rekor_submissions_total, attestor_rekor_submission_latency_seconds, attestor_rekor_submissions_rejected, attestor_rekor_budget_utilization, attestor_rekor_budget_remaining, attestor_rekor_submission_errors, and attestor_rekor_edge_bundle_triggers. None of these instruments exist. Use the names in the table above. Note instruments use a dotted form (attestor.rekor_*); depending on your exporter, OpenTelemetry/Prometheus may normalize dots to underscores at scrape time.

Suggested alerts

These map to instruments that actually exist:

SignalSuggested conditionAction
attestor.rekor_queue_depthsustained high depthInvestigate Rekor reachability / worker health.
attestor.rekor_dead_letter_totalrate increasesInspect dead-letter items; fix root cause; requeue.
attestor.rekor_retry_attempts_totalelevated for a backendBackend degraded; check circuit breaker / failover.

The previously-claimed Grafana dashboard ID stellaops-attestor-rekor could not be verified against any dashboard asset in the repo. Treat it as unverified until a dashboard JSON is added under devops/.

Periodic Verification (related)

Separately from submission, the Attestor ships a scheduled-job implementation (RekorVerificationJob, a Cronos-driven BackgroundService configured by StellaOps.Attestor.Core.Options.RekorVerificationOptions, section Attestor:RekorVerification) that re-verifies logged entries to detect tampering, time-skew, and root-consistency issues. Its metrics are emitted on the separate StellaOps.Attestor.RekorVerification meter (see the Monitoring section above).

Implemented but not wired into the running host (verify before relying on it). As of this revision the Attestor WebService composition (AttestorWebServiceComposition.cs) registers only two hosted services — RekorRetryWorker and (conditionally) TransparencySyncWorker. It does not AddHostedService<RekorVerificationJob>, and there is no production DI registration of IRekorVerificationService or IRekorVerificationStatusProvider (they appear only in tests, the RekorVerificationHealthCheck, and the Doctor plugin). The Doctor check RekorVerificationJobCheck is written to Skip with “Rekor verification service not registered” precisely because the service is absent at runtime. Likewise, the OpenTelemetry WithMetrics block only calls AddMeter(AttestorMetrics.MeterName); the StellaOps.Attestor.RekorVerification meter is not added to the exporter, so its instruments are not scraped by the default pipeline even though they are emitted when the job runs. Treat periodic verification as a code-complete capability that an operator must wire up (host the job + register the service

  • add the meter) before it produces signal. The defaults table below describes the options class as written.

Name-collision gotcha. The codebase has a second class also named RekorVerificationOptions — StellaOps.Attestor.Core.Configuration.RekorVerificationOptions, bound to section Attestor:Rekor — that holds Rekor public-key / checkpoint verification settings (PublicKeyPath, RekorServerUrl, EnableOfflineMode, etc.). It is not the periodic-job options class. The defaults table below is the Core.Options class.

OptionDefaultMeaning
EnabledtrueEnable periodic verification.
CronSchedule"0 3 * * *"Daily at 03:00 UTC.
MaxEntriesPerRun1000Batch cap per run.
SampleRate0.1Fraction of eligible entries verified (1.0 = all).
MaxTimeSkewSeconds300Allowed skew between build time and integratedTime.
LookbackDays90How far back to consider entries (0 = all).
RekorUrlhttps://rekor.sigstore.devServer used for verification.
EnableOfflineVerificationfalseUse stored inclusion proofs without contacting Rekor.
RequireOfflineProofVerificationtrueRecompute Merkle roots for offline proofs.
RequireOfflineCheckpointConsistencytrueBind the proof root to the configured log key.

Air-Gap Considerations

There is no dedicated attestor.rekor_offline_queue table. Air-gap behavior is achieved through two existing mechanisms:

  1. Durable queue holds submissions. When the configured Rekor backend is unreachable, items remain in attestor.rekor_submission_queue in pending / retrying state and drain when connectivity returns. With sufficiently large MaxAttempts, this is the offline buffer. (Note: with the default MaxAttempts = 5 and MaxDelayMs = 60000, items dead-letter after a few minutes of continuous failure — raise MaxAttempts for long disconnections, or requeue dead-lettered items.)
  2. Local transparency log. For fully offline deployments, the Attestor can target a local transparency backend instead of public Rekor; see AttestorOptions.TransparencyOptions.Local (Attestor:Transparency:Local, default origin stellaops.local-transparency.v1, default log URL stellaops://attestor/local-transparency) and LocalTransparencyRekorClient.
  3. Offline verification. RekorVerificationOptions.EnableOfflineVerification verifies stored inclusion proofs without network access. This is an option on the periodic RekorVerificationJob; because that job is not hosted by default (see Periodic Verification), the offline re-verification path only runs once an operator wires the job up — it is not exercised automatically by the running Attestor.

Troubleshooting

Q: Submissions are stuck in retrying / never reach submitted.

Q: Items are landing in dead_letter.

Q: Queue depth grows unbounded.

Not applicable / not implemented. There is no 429 / budget-rejection path for Rekor submissions and no edge-bundle escalation triggers. Troubleshooting guidance referencing 429, budget_remaining, or “edge bundles not submitted” from older drafts does not apply.

Configuration Reference

The Attestor binds AttestorOptions from the Attestor configuration section. Rekor settings live under Attestor:Rekor with per-backend sub-sections (Primary / Mirror) — there is no flat attestor.rekor.serverUrl and no top-level attestor.rekor.enabled/tier/budget. The durable queue binds separately from Attestor:rekorQueue.

Attestor:
  Rekor:
    Primary:
      # Rekor server URL (default: local transparency log,
      # "stellaops://attestor/local-transparency"; set to a real Rekor URL
      # such as https://rekor.sigstore.dev for online use)
      Url: "https://rekor.sigstore.dev"

      # Log version: Auto or V2 (V2 uses tile-based Sunlight format). Default: Auto.
      Version: Auto

      # Tile base URL for v2 (optional; defaults to {Url}/tile/)
      TileBaseUrl: ""

      # Log ID for multi-log environments (hex-encoded SHA-256 of the log key).
      # Production public Rekor:
      LogId: "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"

      ProofTimeoutMs: 15000
      PollIntervalMs: 250
      MaxAttempts: 60

    Mirror:
      Enabled: false
      # ...same RekorBackendOptions fields as Primary...

    # Circuit breaker for resilient Rekor calls
    CircuitBreaker:
      Enabled: true
      FailureThreshold: 5
      SuccessThreshold: 2
      OpenDurationSeconds: 30
      FailureWindowSeconds: 60
      HalfOpenMaxRequests: 3
      UseCacheWhenOpen: true
      FailoverToMirrorWhenOpen: true

  # Durable Rekor submission retry queue (RekorQueueOptions)
  rekorQueue:
    Enabled: true
    MaxAttempts: 5
    InitialDelayMs: 1000
    MaxDelayMs: 60000
    BackoffMultiplier: 2.0
    BatchSize: 10
    PollIntervalMs: 5000
    DeadLetterRetentionDays: 30

  # Periodic transparency-log re-verification (RekorVerificationOptions)
  RekorVerification:
    Enabled: true
    CronSchedule: "0 3 * * *"
    MaxEntriesPerRun: 1000
    SampleRate: 0.1
    MaxTimeSkewSeconds: 300
    LookbackDays: 90
    RekorUrl: "https://rekor.sigstore.dev"
    EnableOfflineVerification: false
    RequireOfflineProofVerification: true
    RequireOfflineCheckpointConsistency: true

Attestor request quotas (separate from Rekor submission)

AttestorOptions.Quotas (section Attestor:Quotas) configures API request throttling for the Attestor service itself — not Rekor submission budgets:

Attestor:
  Quotas:
    PerCaller:
      Qps: 50
      Burst: 100
    Bulk:
      RequestsPerMinute: 6
      MaxItemsPerJob: 100
      MaxQueuedJobs: 20
      MaxConcurrentJobs: 1

Roadmap: Submission Tiers & Budgets

Status: NOT IMPLEMENTED (roadmap / draft). The following model — graph-only vs. edge-bundle tiers, per-tenant hourly/burst/daily submission budgets, and CVSS/policy-gate escalation triggers — has no corresponding code, options, schema, or metrics in the repository as of this revision. It is kept here as a design sketch only. Do not configure these keys; they are ignored.

The envisioned future model:

If/when this is built, it would require: a tier/budget options class, a quota store (the proposed attestor.rekor_quotas table), admission control with a 429/backpressure path on enqueue, and budget/rejected metrics — none of which exist today.