Runbook: Attestor - Signature Generation Failures

Purpose: diagnose and clear failures of the Attestor sign endpoint (POST /api/v1/attestations:sign) so the release pipeline can produce new signed evidence again. Audience: Platform and Security on-call engineers responding to a blocked release, rising signing-failure metrics, or a failing key-material Doctor check.

Reconciled against source 2026-05-31. The Doctor check IDs, metrics, alerts, and CLI commands below were corrected to match src/Attestor and src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Attestor. Several commands the original draft referenced (stella attest test-sign, stella attest config set, stella attest retry, stella keys generate/activate, stella crypto hsm ...) do not exist in the CLI and have been removed or replaced. See “Verified facts” notes.

Caveats verified in source (do not treat as fully live):

  • stella doctor run --plugin <id> is not a real flag — doctor/doctor run accept --check, --category, --tag, --mode, etc., but no --plugin (src/Cli/StellaOps.Cli/Commands/DoctorCommandGroup.cs CreateRunOptions). Use --tag attestation or --category Security to run the Attestor group.
  • check.attestation.keymaterial (SigningKeyExpirationCheck) is defined but is not registered in AttestorDoctorPlugin.GetChecks(), so it does not run via the Attestor Doctor plugin today. Only check.attestation.cosign.keymaterial (+ the three Rekor checks and the transparency-consistency check) are wired.
  • The former unregistered stella keys list|status|rotate|audit sample-data group was deleted on 2026-07-29. stella crypto keys generate remains a separate stub, and stella crypto profile validate targets a Platform endpoint that returns endpoint_not_implemented (exit 0) until RP-028-013 lands. These are forward-looking verbs; verify key health against attestor.signing config + the sign endpoint, not these commands’ output.

Metadata

FieldValue
ComponentAttestor
SeverityCritical
On-call scopePlatform team, Security team
Last updated2026-05-31
Doctor checkscheck.attestation.cosign.keymaterial (registered, Fail). check.attestation.keymaterial exists in source but is not registered in the Attestor plugin (see Caveats).

Symptoms

Verified facts (source ground truth):

  • There is no attestor_signing_failures_total metric and no AttestorSigningFailed alert. Signing failures surface through attestor.sign_total{result="failure"} (src/Attestor/.../Infrastructure/Signing/AttestorSigningService.cs RecordFailureMetrics) and, for unexpected internal errors, attestor.errors_total{type="sign"}. The shipped attestor alerts are AttestorSignLatencyP95High, AttestorVerifyLatencyP95High, AttestorVerifyFailureRate, and AttestorKeyRotationStale (devops/telemetry/alerts/attestation-alerts.yaml).
  • The sign endpoint returns failures as RFC7807 problems with a code extension; the code values are defined in AttestorSigningException usages (src/Attestor/.../Infrastructure/Signing/AttestorSigningService.cs, .../AttestorSigningKeyRegistry.cs).

Impact

Impact TypeDescription
User-facingReleases blocked; new attestations cannot be signed
Data integrityExisting attestations remain valid; only new signing is affected
SLA impactRelease SLO violated; evidence chain cannot advance until signing recovers

Diagnosis

Quick checks

  1. Run the Doctor key-material check:

    stella doctor --check check.attestation.cosign.keymaterial

    Problem if: result is Fail (e.g. “Signing key file not found”, “Signing mode is ‘file’ but KeyPath not configured”, “Signing mode is ‘kms’ but KmsKeyRef not configured”, or “Unknown signing mode”).

  2. Run the full Attestor diagnostic group (filter by tag or category — there is no --plugin flag):

    stella doctor run --tag attestation
# or, broader: stella doctor run --category Security

    The Attestor plugin (stellaops.doctor.attestor, category Security) registers check.attestation.cosign.keymaterial plus the Rekor connectivity / clock-skew / verification-job and transparency-log-consistency checks (AttestorDoctorPlugin.GetChecks()). The key-expiration check check.attestation.keymaterial is not registered today (see Caveats), so it will not appear in this run.

  3. Inspect configured signing keys through the Attestor API/configuration.

    The former unregistered stella keys sample-data group was deleted on 2026-07-29; it never represented live key state. The source of truth is the attestor.signing.keys[] configuration plus the live sign endpoint in Deep diagnosis below. The registered stella key rotation group also fails closed until its service client is configured.

Deep diagnosis

  1. Reproduce the failure against the sign endpoint to capture the exact error code:

    curl -sS --cert client.pem --key client.key \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST https://attestor.internal:8444/api/v1/attestations:sign \
  -d '{"keyId":"<key-id>","payloadType":"application/vnd.in-toto+json","payload":"<base64>","artifact":{"sha256":"<digest>","kind":"image"}}'

    The response problem body carries code. Map it to a cause:

    codeMeaning (source)
    key_missingRequest omitted keyId (AttestorSigningService / AttestorSigningKeyRegistry).
    key_not_foundkeyId is not present in attestor.signing.keys[].
    payload_missing / payload_type_missing / payload_invalid_base64Malformed request body.
    provider_route_driftResolved crypto provider differs from the key’s configured provider.
    provider_key_driftResolved provider key id differs from the key’s providerKeyId.
    provider_algorithm_driftResolved algorithm differs from the key’s configured algorithm.
    signing_failedUnexpected internal error (provider unreachable, key material unloadable); see logs.

  2. Inspect the configured key route in attestor.signing (mode, provider, providerKeyId, algorithm): the drift codes above mean the running crypto provider no longer matches the static configuration — typically a provider/key/algorithm change that was applied to the crypto provider but not to attestor.signing.keys[] (or vice-versa).

  3. Check signing-key expiration (separate Doctor check check.attestation.keymaterial, SigningKeyExpirationCheck, default severity Warn):

    stella doctor --check check.attestation.keymaterial

    By design it fails when a key is past expiry or within 7 days (CriticalDays) and warns within 30 days (WarningDays). Caveat: this check is not registered in the Attestor Doctor plugin today (it is omitted from AttestorDoctorPlugin.GetChecks()), and its key inventory (GetSigningKeysAsync) returns demo data rather than the live key store — so it may report “Skip”/sample results or not resolve at all. For an expired key, rely on the attestor.signing config and follow attestor-key-expired.md.

  4. If the provider is a remote HSM/KMS, confirm reachability of the backing service before blaming Attestor. See attestor-hsm-connection.md for the HSM/KMS connectivity path. Stella Ops is on-prem first: the default signing provider is the File-KMS provider (signing.kms with a local rootPath) or a HashiCorp Vault-backed provider — there are no cloud-managed KMS defaults.

Resolution

Immediate mitigation

  1. If the configured key route drifted (provider_route_drift / provider_key_drift / provider_algorithm_drift): align attestor.signing.keys[] with the active crypto provider, or point signing at a known-good key by setting the request’s keyId to a healthy entry. Restart the Attestor service so it reloads attestor.signing (there is no live reload command — Attestor reads options at startup).

  2. If key material is missing/unreadable (Doctor check.attestation.cosign.keymaterial = Fail for file mode): restore the key file and repoint the configuration, then restart the service. Note two distinct config surfaces: the Doctor check keys off the flat Attestor:Signing:Mode / Attestor:Signing:KeyPath / Attestor:Signing:KmsKeyRef values (recognized modes keyless | file | kms, else “Unknown signing mode”), whereas the runtime signing registry loads per-key material from attestor.signing.keys[].MaterialPath (or inline Material). The Doctor “file”/KeyPath model and the registry’s per-key MaterialPath are not the same setting — reconcile both when repointing.

  3. If the active key is expired, switch signing to a healthy key in attestor.signing.keys[] and follow attestor-key-expired.md for rotation.

Verified facts: There is no stella attest config set signing.mode software, stella attest reload, stella attest retry, stella attest test-sign, or stella attest status (src/Cli/StellaOps.Cli/Commands/AttestCommandGroup.cs exposes build, attach, verify, verify-offline, list, fetch, predicates; fix-chain/patch are sibling command groups, not subcommands of attest). Signing mode is per-key in attestor.signing.keys[].mode, not a global software/HSM toggle. In the runtime registry the mode defaults to kms (for KMS keys) or keyful otherwise (AttestorSigningKeyRegistry.CreateEntry); the broader signing-mode vocabulary recorded in bundle/watchlist metadata is keyless | kms | hsm | fido2 (AttestationBundle.cs, AttestorEntryInfo.cs). The registry itself dispatches on provider (default/ECDSA, bouncycastle.ed25519, cn.sm.soft/SM2, kms/File-KMS); there is no dedicated hsm provider path in AttestorSigningKeyRegistry. Configuration is applied via the YAML/config file and picked up on restart.

Root cause fix

Provider drift (key route mismatch):

  1. Reconcile the crypto provider with the per-key configuration in attestor.signing.keys[] (provider, providerKeyId, algorithm, kmsVersionId). Confirm the active crypto provider with:

    stella crypto status
stella crypto provider show

  2. Validate the active crypto profile (intended to catch plugin-reachability / FIPS-mode disagreements):

    stella crypto profile validate

    Roadmap: the validate verb exists, but it calls the Platform endpoint GET /api/v1/admin/crypto/profile/validate, which currently returns an endpoint_not_implemented payload with exit code 0 until RP-028-013 lands a real verdict (CryptoCommandGroup.BuildProfileCommand). Adopt it in CI now, but do not rely on it as a live gate yet.

Missing or rotated key material:

  1. Generate a key pair with cosign-compatible tooling and register it in attestor.signing.keys[]:

    cosign generate-key-pair --output-key-prefix stellaops
# stella crypto keys generate exists but currently only prints a success message
# (CryptoCommandGroup.BuildKeysCommand stub) — it does not yet write real key files.

  2. Update attestor.signing.keys[] to reference the new key (id, provider, algorithm, material path), then restart Attestor.

Key rotation (preferred ongoing fix):

Use the Signer key-rotation HTTP API described in the module runbook. The former stella keys rotate sample-data handler was unregistered and was deleted on 2026-07-29. The registered singular stella key rotate command still fails closed until its real IKeyRotationService client is configured; it must not be treated as an applied rotation.

Verification

# 1. Confirm key material and route are healthy
stella doctor --check check.attestation.cosign.keymaterial

# 2. Re-issue a sign request and confirm HTTP 200 with a bundle + key metadata
curl -sS --cert client.pem --key client.key \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -X POST https://attestor.internal:8444/api/v1/attestations:sign \
  -d '{"keyId":"<key-id>","payloadType":"application/vnd.in-toto+json","payload":"<base64>","artifact":{"sha256":"<digest>","kind":"image"}}'

# 3. Confirm the failure metric has stopped climbing (Prometheus)
#    attestor.sign_total{result="failure"}  and  attestor.errors_total{type="sign"}

Prevention