Runbook: Attestor - Attestation Verification Failures

Purpose: triage attestation verification failures that block a promotion or release — signature/body mismatches, transparency-log inconsistencies, trust/key-rotation gaps, clock skew, and unreachable logs. Audience: Platform and Security on-call engineers responding to a blocked stella verify release chain step or a failing POST /api/v1/rekor/verify.

Doc-vs-code reconciliation (2026-05-31): This runbook was reconciled against src/Attestor (esp. StellaOps.Attestor.WebService, StellaOps.Attestor.Core/Verification, and StellaOps.Attestor.Infrastructure/Verification), the Doctor plugin src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Attestor, and the CLI command tree under src/Cli/StellaOps.Cli/Commands. Several command names and request/response shapes from earlier drafts did not match the code and have been corrected or flagged. Commands tagged (roadmap) are not implemented today; use the documented alternative. Items tagged (not wired) describe a type that exists in the codebase but is not registered in the running Attestor WebService.

Metadata

Field	Value
Component	Attestor
Severity	High
On-call scope	Platform team, Security team
Last updated	2026-05-31
Live verify API	`POST /api/v1/rekor/verify` (`AttestorWebServiceEndpoints`); service `AttestorVerificationService` (registered in `AddAttestorInfrastructure`)
Doctor check	`check.attestation.rekor.verification.job` (Doctor plugin `StellaOps.Doctor.Plugin.Attestor`; skips unless the verification status provider is registered — see note)
Verify scope	ASP.NET policy `attestor:verify`; accepts scope-claim values `attestor.verify` or `attestor.write` (`AttestorWebServiceComposition`)

Scope catalog gotcha. The Attestor authorization policies (attestor:write, attestor:verify, attestor:read) and the scope-claim values they accept (attestor.write / attestor.verify / attestor.read) are defined locally in AttestorWebServiceComposition.cs. They are not present in the canonical scope catalog StellaOps.Auth.Abstractions/StellaOpsScopes.cs (which carries the separate attest:create / attest:read / attest:admin consts). attestor:read is satisfied by any of attestor.read/attestor.verify/attestor.write; attestor:verify by attestor.verify/attestor.write; attestor:write by attestor.write. Trusted-network service bypass also satisfies all three.

Note on health / Doctor surfaces. Two related-but-distinct subsystems exist in the code, and only one is wired into the running service:
Live verify API — POST /api/v1/rekor/verify is backed by IAttestorVerificationService → AttestorVerificationService, which is registered. This is the surface to use for on-demand verification (Diagnosis).
Periodic Rekor verification job + health check + status provider (RekorVerificationJob, RekorVerificationHealthCheck with Name = "rekor-verification", IRekorVerificationStatusProvider) exist in StellaOps.Attestor.Core.Verification but are (not wired) into the Attestor WebService: its AddHealthChecks() registers only the self check, and the status provider / background job have no production DI registration. The Doctor plugin check check.attestation.rekor.verification.job therefore Skips with “Rekor verification service not registered” until those services are wired. Do not expect a rekor-verification entry on the health endpoint today.

Symptoms

[ ] Attestation verification failing
[ ] Alert AttestorVerificationFailed firing (operator-defined; the platform does not ship this alert rule by default)
[ ] POST /api/v1/rekor/verify returns a result with ok=false and a populated issues[] list (free-form strings such as time_skew_rejected: ... plus any engine-reported issues), or returns 400 Bad Request with an AttestorVerificationException code (e.g. not_found, invalid_query)
[ ] Periodic Rekor verification job reporting failures — only if the job is wired (see Metadata note). Its RekorVerificationFailureCode values are: EntryNotFound, InvalidSignature, InvalidInclusionProof, TimeSkewExceeded, BodyHashMismatch, LogIndexMismatch, NetworkError, Timeout, Unknown. These are distinct from the live verify-API issues[] strings above.
[ ] Promotions/releases blocked because a stella verify release chain step failed

Impact

Impact Type	Description
User-facing	Artifacts cannot be promoted; release blocked
Data integrity	May indicate tampered attestation, transparency-log inconsistency, or configuration issue
SLA impact	Release pipeline blocked until resolved

Diagnosis

Quick checks

Check the Attestor service health:
```
# The Attestor WebService maps two health endpoints (AttestorWebServiceComposition):
curl -s https://<attestor-host>/health/ready
curl -s https://<attestor-host>/health/live
```
These use the default ASP.NET health writer and return a plain status string (Healthy / Degraded / Unhealthy), not a JSON entries[...] document. Today the only registered check is self; the rekor-verification check (not wired) does not appear on these endpoints. The RekorVerificationHealthCheck logic — were it registered — would report data keys lastRunStatus, entriesVerified, entriesFailed, failureRate, rootConsistent, criticalAlerts, returning Healthy when verification is disabled, Degraded when it has never run or is stale (>48h), and Unhealthy on critical alerts, failure rate over threshold, or a failed root-consistency check. Until it is wired, use the live verify API below.
Verify a specific attestation against the transparency log (API):
```
# ASP.NET policy attestor:verify — accepts scope-claim attestor.verify or attestor.write.
curl -s -X POST https://<attestor-host>/api/v1/rekor/verify \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "uuid": "<rekor-uuid>", "refreshProof": true }'
```
The request body is an AttestorVerificationRequest: supply at least one of uuid, bundle, or artifactSha256. refreshProof: true re-pulls the proof from the log; offline: true skips contacting external logs. Returns an AttestorVerificationResult — inspect ok (bool), status, and issues[] (camelCase). A 400 carries an AttestorVerificationException code (e.g. not_found, invalid_query).
Verify a DSSE / evidence bundle offline (CLI):
```
stella attest verify-offline --bundle <bundle.tar.gz> \
  --trust-root <trust-root-dir> \
  --checkpoint <checkpoint.sig> \
  --strict --format json
```
stella attest verify-offline validates a bundle’s manifest integrity, DSSE envelope signatures, Rekor inclusion proof (optional unless --strict), and content hash without network access. --trust-root/-r points at a directory containing CA certs and the Rekor public key; --checkpoint/-c and --artifact/-a are optional. (There is no stella attest verify --envelope --root --explain form: stella attest verify is the OCI-image verifier and takes --image/-i; offline bundle verification is verify-offline.)

Deep diagnosis

Inspect attestation details (API):
```
# ASP.NET policy attestor:read (accepts attestor.read/attestor.verify/attestor.write).
curl -s "https://<attestor-host>/api/v1/attestations/<uuid>?refresh=true" \
  -H "Authorization: Bearer $TOKEN" | jq
```
The detail response (AttestationDetailResponseDto, camelCase) carries uuid, index, backend, proof (inclusion proof / checkpoint), logURL, status, localTransparencyReceipt, externalMirrorReceipts, mirror, and artifact. Note: this endpoint does not include a signer field — signer identity (signer.{mode,issuer,subject,keyId}) is returned by the list endpoint (GET /api/v1/attestations), not the per-UUID detail. The equivalent Rekor alias is GET /api/v1/rekor/entries/<uuid> (same handler).
List attestations for an artifact (CLI):
```
# Both subcommands require --image; fetch also requires --predicate-type.
stella attest list --image <registry/repo@sha256:...>
stella attest fetch --image <registry/repo@sha256:...> --predicate-type <uri>
```
(These CLI subcommands are OCI-registry oriented and are currently placeholder implementations — attest list/attest verify carry TODO: Integrate with IOciAttestationAttacher in the source. Treat their output as illustrative until wired; the API endpoints above are the authoritative surface.)
Check the active crypto provider / plugin reachability:
```
stella crypto status
stella crypto plugins status
stella crypto profile validate
```
stella crypto profile validate reports a plugin-reachable / FIPS-mode disagreement, a common cause of “algorithm not supported” verification failures.
Check trust state (TUF):
```
stella trust status               # current trust state + TUF metadata freshness
stella trust status --show-keys   # include loaded key fingerprints
stella trust verify <artifact>    # verify an artifact against TUF-loaded anchors
```
stella trust verify takes a positional artifact reference (image ref, file path, or attestation) and supports --check-inclusion (default on) and --offline. (There is no stella trust-anchors command — see the roadmap note below.)

Resolution

Earlier drafts listed several stella commands / flags that do not exist in the CLI (stella trust-anchors ..., stella trust init --repo, stella attest show/create/integrity-check/resign, stella attest verify --envelope --root --explain, stella verify cert-chain, stella keys show, stella issuer trust-status/show/trust, stella issuer keys fetch, stella crypto providers list --algorithms). The real surfaces are: offline bundle verification is stella attest verify-offline (--bundle/--trust-root/--checkpoint); TUF init is stella trust init --tuf-url; provider info is stella crypto provider show / stella crypto profiles list. They are used below, or flagged (roadmap) where no equivalent exists yet.

Immediate mitigation

Refresh trust metadata (TUF) so the verifier sees current roots/keys:
```
stella trust sync
stella trust status
```

Re-verify after a trust refresh:

stella attest verify-offline --bundle <bundle.tar.gz> --trust-root <trust-root-dir> --strict

Or against the transparency log:

curl -s -X POST https://<attestor-host>/api/v1/rekor/verify \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{ "uuid": "<rekor-uuid>", "refreshProof": true }'

Bootstrap a TUF trust repository if none is initialized:

# --tuf-url/-u is the repository URL (required); there is no --repo flag.
stella trust init --tuf-url <trust-repo-url>

(roadmap) Add an individual issuer certificate / intermediate to a trust anchor list. There is no stella trust-anchors add / stella trust-anchors add-intermediate / stella trust-anchors add-key command. Trust roots are managed via the TUF trust repository (stella trust init|sync|export|import) and the crypto provider configuration. Use stella trust sync to pull the current roots, or update the TUF repo upstream.

Root cause fix

Match the failure to what the verify surface reports. Note the two distinct vocabularies (see Metadata note):

Live verify API (POST /api/v1/rekor/verify): a 400 carries an AttestorVerificationException code (not_found, invalid_query, plus any engine code), and issues[] carries free-form strings (e.g. time_skew_rejected: ...) produced by the verification engine.
Periodic job (RekorVerificationFailureCode, only if wired): enum values InvalidSignature, BodyHashMismatch, InvalidInclusionProof, LogIndexMismatch, TimeSkewExceeded, EntryNotFound, NetworkError, Timeout, Unknown.

The headings below use the periodic-job enum names as a familiar shorthand for the failure class; map them to the engine issues[] strings you actually see.

InvalidSignature / BodyHashMismatch (signature or body mismatch):

Confirm the envelope/body was not modified by re-fetching from the log and re-verifying offline:

curl -s "https://<attestor-host>/api/v1/attestations/<uuid>?refresh=true" \
  -H "Authorization: Bearer $TOKEN" | jq '.proof, .artifact'
stella attest verify-offline --bundle <bundle.tar.gz> --trust-root <trust-root-dir> --strict

If the artifact legitimately changed, re-sign and re-submit a fresh attestation:

# Sign a new attestation payload (requires attestor:write).
curl -s -X POST https://<attestor-host>/api/v1/attestations:sign \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d @sign-request.json
# Then submit to the transparency log:
curl -s -X POST https://<attestor-host>/api/v1/rekor/entries \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d @submission.json

(There is no stella attest integrity-check, stella attest create, or stella attest resign command — use the :sign and /rekor/entries APIs above.)

Key rotated and old key not trusted:

Refresh the TUF trust state so rotated keys/roots are visible to the verifier:
```
stella trust sync
stella trust status
```
Inspect or rotate issuer keys via the issuer key lifecycle commands:
```
stella issuer keys list
stella issuer keys rotate <key-id>
```
(There is no stella issuer keys fetch or stella issuer trust command; stella issuer keys provides list, create, rotate, and revoke.)

Certificate expired / chain invalid:

Validate the active crypto profile and plugin reachability:
```
stella crypto profile validate
stella crypto status
```
Re-verify offline with the trust-root chain supplied explicitly:
```
stella attest verify-offline --bundle <bundle.tar.gz> --trust-root <chain-dir> --strict
```
(There is no stella verify cert-chain or stella verify cert command, and stella attest verify has no --explain flag; chain/expiry diagnostics surface in the verify-offline per-check report and in the API result issues[].)

InvalidInclusionProof / LogIndexMismatch / root inconsistency:

Re-fetch the entry with refresh=true to pull the latest inclusion proof and checkpoint:

curl -s "https://<attestor-host>/api/v1/attestations/<uuid>?refresh=true" \
  -H "Authorization: Bearer $TOKEN" | jq '.proof'

A persistently failing root-consistency check (the periodic job’s RekorVerificationHealthCheck logic reports "Rekor root consistency check failed
- possible log tampering", and the Doctor check check.attestation.rekor. verification.job fails) is a security event — escalate to the Security team before re-submitting. Note both surfaces are (not wired) today (see Metadata); the live API exposes inclusion-proof issues directly in issues[].

TimeSkewExceeded:

The verifier enforces a maximum timestamp skew (TimeSkewValidator; controlled by Attestor options TimeSkew.*). When the checkpoint/integrated time is outside tolerance, the live API surfaces it as an issues[] entry prefixed time_skew_rejected: (when TimeSkew.FailOnReject is set). Check host/NTP clock sync and the configured skew tolerance, then re-verify:
```
curl -s -X POST https://<attestor-host>/api/v1/rekor/verify \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{ "uuid": "<rekor-uuid>", "refreshProof": true }' | jq '.ok, .issues'
```
(There is no --require-timestamp / --max-skew CLI flag on stella attest verify; skew tolerance is server-side configuration.)

EntryNotFound / NetworkError / Timeout (transparency log unreachable):

The entry may be valid locally but not anchored, or Rekor may be unreachable. See the companion runbook attestor-rekor-unavailable.md.

Algorithm not supported:

Confirm the active provider supports the attestation’s algorithm:
```
stella crypto status
stella crypto plugins status
```
(There is no stella crypto providers (plural) command and no --algorithms flag. The singular stella crypto provider show reports only the installation provider region setting; the available profiles and their algorithm sets are listed by stella crypto profiles list. Provider/plugin reachability is reported by stella crypto plugins status and stella crypto profile validate.)

Verification

# Verify the entry against the transparency log (API; policy attestor:verify).
curl -s -X POST https://<attestor-host>/api/v1/rekor/verify \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{ "uuid": "<rekor-uuid>", "refreshProof": true }' | jq '.ok, .status, .issues'

# Verify an evidence bundle offline against trust roots.
stella attest verify-offline --bundle <bundle.tar.gz> --trust-root <trust-root-dir> --strict

# Verify a full release promotion chain (source, build, signature, transparency).
stella verify release <release-bundle>

# Re-check service health (plain status string; no rekor-verification entry today).
curl -s https://<attestor-host>/health/ready

Prevention

[ ] Trust metadata: Keep the TUF trust repository current; run stella trust sync on a schedule
[ ] Key rotation: Plan issuer-key rotation with an overlap period (stella issuer keys rotate --overlap-days) for verification continuity
[ ] Monitoring: Once the periodic Rekor verification job and its health check are wired (currently not wired — see Metadata), alert on its failureRate, rootConsistent, and criticalAlerts. Until then, monitor verify-API outcomes and the check.attestation.rekor.verification.job Doctor check
[ ] Testing: Include stella verify release / stella attest verify-offline in the release pipeline
[ ] Crypto profile: Run stella crypto profile validate after provider/profile changes

Architecture: docs/modules/attestor/rekor-verification-design.md, docs/modules/attestor/transparency.md, docs/modules/attestor/api-reference.md (the previously linked docs/modules/attestor/verification.md does not exist)
Related runbooks: attestor-signing-failed.md, attestor-key-expired.md, attestor-rekor-unavailable.md
Trust management: docs/operations/trust-lattice-runbook.md, docs/operations/trust-lattice-troubleshooting.md, docs/operations/rekor-policy.md (the previously linked docs/operations/trust-anchors.md does not exist)