Runbook: Attestor - Rekor Transparency Log Unreachable

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-005 - Attestor Runbooks Reconciled against src/Attestor, src/Doctor, and src/Cli on 2026-05-31 (options, metrics, Doctor check IDs, WebService scopes/endpoints, and the attestor transparency CLI verbs verified against source).

Queue currency note (2026-07-17). Current publication persists locally first and records optional external-sync attempts in durable mirror receipts. RekorRetryWorker and attestor.rekor_submission_queue are legacy drain compatibility only; no current submission producer enqueues them. Diagnose a new external outage through transparency target status/receipts and the sync runner. Use queue-depth/worker steps later in this runbook only for rows left by an older release.

Read this first — what “Rekor” means in Stella Ops. By default the Attestor’s primary transparency anchor is a Stella-owned local transparency log, not the public Sigstore instance. In AttestorOptions (src/Attestor/.../Core/Options/AttestorOptions.cs) the default is Rekor.Primary.Url = "stellaops://attestor/local-transparency" (LocalTransparencyOptions.DefaultLogUrl). This is deliberate for the offline/air-gap-first posture: a sealed site is fully functional with no external Rekor reachable. Public Sigstore Rekor (https://rekor.sigstore.dev) is only contacted when an operator explicitly points Rekor.Primary.Url (or a mirror) at it, or when the Doctor check.attestation.rekor.connectivity probe runs with its own fallback endpoint. Treat “Rekor unreachable” as “the configured external transparency endpoint is unreachable” — local anchoring keeps working regardless.

Metadata

FieldValue
ComponentAttestor
SeverityHigh (degraded to Medium when local transparency is the primary anchor)
On-call scopePlatform team
Last updated2026-05-31
Doctor checkcheck.attestation.rekor.connectivity
Related Doctor checkscheck.attestation.rekor.verification.job, check.attestation.transparency.consistency, check.attestation.clock.skew
Health checkrekor-verification (ASP.NET IHealthCheck, see RekorVerificationHealthCheck)

Symptoms

NOTE: There is no alert named AttestorRekorUnavailable shipped in the repository. Wire your own Prometheus alert on the metrics above (attestor.rekor_queue_depth, attestor.rekor_dead_letter_total, attestor.rekor_submission_status_total) — see Prevention.

Impact

Impact TypeDescription
User-facingAttestations not anchored to the external transparency log; local transparency receipts remain authoritative
Data integrityAttestations are still signed and locally anchored; only external/public transparency is reduced
SLA / complianceIf a compliance regime requires external transparency anchoring, that requirement is degraded until the endpoint recovers
VerificationIf requireTransparencyInclusion is enabled, verification of new entries can fail until inclusion proofs land

Diagnosis

Quick checks

  1. Run the Doctor connectivity check:

    stella doctor --check check.attestation.rekor.connectivity

    The check probes {rekorUrl}/api/v1/log, reports latency and tree size on success, and returns Skip in sealed/air-gap mode. The endpoint it probes is resolved from the Attestor:Rekor:Url config key, falling back to Transparency:Rekor:Url, then https://rekor.sigstore.dev.

    Key-name gotcha (verified in source): the Doctor probe key Attestor:Rekor:Url is not the same key the Attestor service binds for its submission anchor, which is Attestor:Rekor:Primary:Url (AttestorOptions.RekorOptions.Primary.Url). The service does not read a flat Attestor:Rekor:Url. Consequence: if you only set Attestor:Rekor:Primary:Url, the connectivity/clock-skew/consistency checks still fall through to https://rekor.sigstore.dev and may report Fail even though the service is healthily anchoring elsewhere (e.g. the local log). Point Attestor:Rekor:Url (or Transparency:Rekor:Url) at the same endpoint you want the Doctor checks to probe. (RekorConnectivityCheck, RekorClockSkewCheck, and TransparencyLogConsistencyCheck all read the flat key.)

  2. Check the periodic verification job health:

    stella doctor --check check.attestation.rekor.verification.job

  3. Inspect the effective Attestor configuration (read-only):

    stella config show attestor    # 'config' is inspect-only: list / show <path>

    Confirm what attestor.rekor.primary.url and attestor.rekor.mirror.* are actually set to. If primary.url is still the default stellaops://attestor/local-transparency, external Rekor is not in the submission path and this incident is most likely about the verification probe or an explicitly configured mirror.

Deep diagnosis

  1. Determine the configured primary/mirror anchor. In service config (appsettings.json root key attestor, or env vars):

    • attestor:rekor:primary:url
    • attestor:rekor:mirror:enabled / attestor:rekor:mirror:url
    • attestor:rekor:circuitBreaker:* (enabled by default; opens after failureThreshold=5 failures within failureWindowSeconds=60)

  2. Manually probe the configured endpoint (replace with the actual URL):

    curl -s https://rekor.example.com/api/v1/log | jq .
nc -zv rekor.example.com 443

  3. Check the Attestor service logs for RekorRetryWorker activity. The worker logs queue depth (pending, submitting, retrying, dead_letter) and per-item submission failures. Look for repeated Rekor queue item ... submission failed warnings and exceeded max attempts ... moved to dead letter errors.

  4. Inspect metrics (Meter StellaOps.Attestor):

    • attestor.rekor_queue_depth (observable gauge) — current pending + retrying
    • attestor.rekor_retry_attempts_total{backend,attempt}
    • attestor.rekor_submission_status_total{status,backend}
    • attestor.rekor_dead_letter_total{backend}
    • attestor.submit_total{result,backend} / attestor.submit_latency_seconds

  5. List recent attestations and their log/mirror status via the API to see which entries lack external mirror receipts:

    GET /api/v1/attestations           # requires attestor.read scope
GET /api/v1/rekor/entries/{uuid}   # set ?refresh=true to bypass cache

Scope note: the attestor HTTP endpoints authorize on the scope claim values attestor.read, attestor.verify, attestor.write (dot form, see AttestorWebServiceComposition). The ASP.NET authorization policy names are the colon forms attestor:read / attestor:verify / attestor:write — do not confuse the two.

Resolution

Immediate mitigation

  1. Rely on the durable retry queue (default behavior). A failed external submission is not lost — RekorRetryWorker retries from the durable queue with exponential backoff. The queue is governed by RekorQueueOptions (config root attestor; defaults: enabled=true, maxAttempts=5, initialDelayMs=1000, maxDelayMs=60000, backoffMultiplier=2.0, pollIntervalMs=5000, deadLetterRetentionDays=30). No operator action is required for retries to occur; the worker drains the queue automatically when the endpoint recovers.

  2. Let the circuit breaker fail over (default behavior). RekorCircuitBreakerOptions is enabled by default with failoverToMirrorWhenOpen=true and useCacheWhenOpen=true. If a mirror is configured (attestor:rekor:mirror:enabled=true), traffic fails over to it while the primary circuit is open.

  3. Relax the inclusion requirement (only if compliance allows). To stop verification from failing on entries that have no inclusion proof yet, set attestor:verification:requireTransparencyInclusion=false (and/or attestor:verification:requireCheckpoint=false) and restart the service. Warning: this reduces transparency guarantees; revert once the log recovers. (These are the real knobs in AttestorOptions.VerificationOptions; there is no rekor.required flag.)

  4. Configure or enable a mirror if you have a second Rekor-compatible endpoint, by setting attestor:rekor:mirror:enabled=true and attestor:rekor:mirror:url=... and restarting the service.

The legacy commands stella attest config set/get, stella attest reload, stella attest rekor status|ping|pending-entries|process-queue|test-submit, and the flags rekor.queue_on_failure / rekor.required / rekor.url / rekor.proxy do not exist. Configuration changes are applied through service config (appsettings/env) plus a restart; retries are handled by the background worker, not a manual process-queue command.

Root cause fix

If this is a public/external Rekor outage:

  1. Confirm the outage independently (e.g. Sigstore status, your mirror’s status page). The platform is offline-first — a public Rekor outage does not block signing or local anchoring.
  2. Do nothing destructive. Queued submissions retry automatically; once the endpoint is healthy the worker drains the queue. Watch attestor.rekor_queue_depth trend back to zero.

If this is a network/firewall issue:

  1. Verify outbound HTTPS to the configured endpoint:
    curl -v https://<configured-rekor-host>/api/v1/log --max-time 30
nc -zv <configured-rekor-host> 443
  2. Add the configured Rekor endpoint(s) to your egress allowlist. If you also use Sigstore keyless signing, Fulcio (fulcio.sigstore.dev:443) is needed for certificate issuance in addition to the Rekor host.
  3. There is no built-in HTTP-proxy config key for Rekor; configure egress at the host/network level (standard HTTPS_PROXY/NO_PROXY env handling applies to the .NET HTTP client).

If this is a TLS/certificate issue:

  1. Inspect the handshake:
    openssl s_client -connect <configured-rekor-host>:443 -brief
  2. Update the host CA trust store through your OS/base-image mechanism. (There is no stella crypto ca update command — stella crypto covers sign/verify/profile operations, not CA bundle management.)

If the periodic verification job is failing (root-consistency / clock skew):

  1. A Rekor root consistency check failed from rekor-verification indicates possible log tampering or a checkpoint mismatch — escalate, do not silence. Cross-check check.attestation.transparency.consistency.
  2. A high attestor.time_skew_detected_total or check.attestation.clock.skew failure points at host clock drift; fix NTP before re-running verification.

If this is a sealed / air-gap deployment:

  1. The connectivity check returns Skip when air-gapped — external Rekor probes are intentionally not made. Use the offline transparency exchange flow instead of an external submission. exchange and sync are command groups; the runnable leaves are:
    # Sealed-site controlled-media exchange (create -> validate -> apply):
stella attestor transparency exchange schedule create ...
stella attestor transparency exchange relay-import create ...
stella attestor transparency exchange import validate ...
stella attestor transparency exchange import apply ...

# Connected-window mirror sync (must be enabled — see below):
stella attestor transparency sync run ...

# Inspect targets / receipts / sync reports:
stella attestor transparency targets list
stella attestor transparency receipts list
    Connected-window sync is gated by attestor:transparency:externalSync:enabled (and a separate workerEnabled flag) and exposed via POST /api/v1/transparency/external-sync:run (requires attestor:write; returns a disabled-state problem document if the feature flag is off).

Verification

# Re-run the connectivity check (expect Pass, or Skip when air-gapped)
stella doctor --check check.attestation.rekor.connectivity

# Re-run the verification-job check
stella doctor --check check.attestation.rekor.verification.job

# Confirm the queue is draining (depth trending to zero) via your metrics backend:
#   attestor.rekor_queue_depth, attestor.rekor_dead_letter_total

# Inspect a specific entry, forcing a fresh fetch (requires attestor.read scope)
GET /api/v1/rekor/entries/{uuid}?refresh=true

Prevention

Doc gap (flag, not invented): the prior runbook linked docs/modules/attestor/transparency-log.md and docs/operations/rekor-configuration.md; neither exists in this docs tree. They have been re-pointed to docs/modules/attestor/transparency.md and the module dossier above. If a dedicated Rekor-configuration page is desired, create it under docs/modules/attestor/operations/.