Runbook: Attestor - Signing Key Expired

Purpose: restore attestation signing after a Stella Ops signing key has expired (or is about to), without breaking verification of evidence that was already signed. Audience: Platform and Security on-call engineers responding to a blocked release or a failing key-material Doctor check.

Metadata

FieldValue
ComponentAttestor (Signer)
SeverityCritical
On-call scopePlatform team, Security team
Last updated2026-05-31
Doctor checkcheck.attestation.keymaterial

Source-of-truth note (reconciled 2026-05-31): This runbook was reconciled against src/Attestor, src/Cli, and src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Attestor. Several stella CLI commands referenced in earlier drafts do not exist in the shipped CLI; they are flagged inline below. Production key rotation is driven by the Signer service trust-anchor API (/api/v1/anchors/...). There are two CLI surfaces, and neither is a live operator tool today:

  • stella key (singular) — wired into the root CLI (CommandFactory.BuildKeyCommand → Commands/Proof/KeyRotationCommandGroup.cs). It mirrors the Signer trust-anchor API with subcommands list, add, revoke, rotate, status, history, verify, each taking a trust-anchor GUID argument. However, every subcommand currently fails closed: it logs the request and returns Error: key rotation <op> requires a configured key rotation service. No simulated trust-anchor change or validity result was produced. (exit code ProofExitCodes.SystemError). The API-client wiring is not yet present, so use the Signer HTTP API directly until it is.
  • stella keys (plural) — the unregistered sample-data group was deleted on 2026-07-29. It is not an operator command. Do not confuse it with the registered, fail-closed stella key group or with stella crypto keys.

Commands flagged NOT IMPLEMENTED are called out in the relevant sections below.

Symptoms

Note: The Attestor’s own DSSE verification (AttestationChainValidator / KeyRotationService.CheckKeyValidityAsync) treats a key as Expired for a signature only when signedAt >= ExpiresAt. Existing attestations signed before expiry remain verifiable — see Impact below.

Impact

Impact TypeDescription
User-facingNo new attestations can be signed; releases blocked at signing
Data integrityExisting attestations signed before the key’s expiry remain valid; only new signing is blocked
SLA impactRelease SLO violated; compliance posture compromised

Diagnosis

Quick checks

  1. Run the Doctor key-material check:

    stella doctor run --check check.attestation.keymaterial

    Classification thresholds (from SigningKeyExpirationCheck):

    • Expired (daysUntilExpiry < 0) → Fail
    • Critical (< 7 days) → Fail
    • Warning (< 30 days) → Warn
    • Otherwise → Pass

    For keyless signing mode the check returns Skip (no expiring key material).

  2. Inspect the configured signing mode and key material: The signing mode is read from configuration by the CosignKeyMaterialCheck (these keys are the check’s inputs — there is no bound AttestorSigning options class in src/Attestor; the check falls back to the unprefixed Signing:* keys, then to keyless):

    • Attestor:Signing:Mode — keyless (default), file, or kms
    • Attestor:Signing:KeyPath — file-key path (mode file)
    • Attestor:Signing:KmsKeyRef — KMS key reference (mode kms)
    # Verify key material availability (existence + mode), not expiration:
stella doctor run --check check.attestation.cosign.keymaterial

  3. List and check signing keys (CLI): The wired CLI surface is stella key (singular). It maps to the Signer trust-anchor API and takes the anchor GUID as an argument, but currently fails closed (“requires a configured key rotation service”; see the source-of-truth note above), so it is informational only today:

    stella key list <anchorId>     # --include-revoked, --output text|json
stella key status <anchorId>   # rotation status + warnings

    The former plural stella keys sample-data group was not wired and was deleted on 2026-07-29; it is not an alternate inspection path.

Deep diagnosis

  1. Query the Signer trust-anchor key state (production path): Key lifecycle is owned by the Signer service. Use the trust-anchor API (StellaOps.Signer.WebService → KeyRotationEndpoints), which requires the signer:rotate scope. The endpoint group enforces the KeyManagement authorization policy (KeyRotationEndpoints.MapKeyRotationEndpoints → .RequireAuthorization("KeyManagement")). KeyManagement is a legacy alias of SignerPolicies.KeyManagement ("Signer.KeyManagement"); both are registered in Program.cs and both map to StellaOpsScopes.SignerRotate (signer:rotate):

    # Full key history for an anchor (added/revoked/expires timestamps):
GET /api/v1/anchors/{anchorId}/keys/history

# Rotation warnings (ExpiryApproaching / LongLived / AlgorithmDeprecating):
GET /api/v1/anchors/{anchorId}/keys/warnings

# Whether a key was valid at a given signing time (status: Active/Expired/Revoked/NotYetValid):
GET /api/v1/anchors/{anchorId}/keys/{keyId}/validity?signedAt=<iso8601>

    Signer KeyRotationOptions defaults: ExpiryWarningDays = 60, MaxKeyAgeDays = 365. Deprecated algorithms warned by default: RSA-2048, SHA1-RSA.

  2. Review the key audit trail: Production path — the Signer key history (GET /api/v1/anchors/{anchorId}/keys/history), which is backed by the audited KeyAuditLog (operations Add / Revoke). The wired CLI equivalent is:

    stella key history <anchorId> [--key-id <id>] [--limit 50] [--output text|json]

    (Currently fail-closed — see source-of-truth note.) Use the Signer history endpoint until the registered CLI client is wired.

NOT IMPLEMENTED (removed from earlier draft): the following commands have no implementation in src/Cli: stella keys list --type signing --show-expiration, stella keys show <id> --details, stella crypto cert verify-chain --key <id>, stella keys list --type signing --status inactive, stella keys rotation-history. Use the Signer trust-anchor API (/api/v1/anchors/.../keys/history and .../validity) instead.

Resolution

Immediate mitigation

The Signer key model revokes/rotates keys rather than “activating a backup”. A key removed from an anchor’s AllowedKeyIds stays valid for historical proofs but cannot sign new material.

  1. Rotate the expired/critical key with an overlap period (CLI): The wired command is stella key rotate (singular), which takes the trust-anchor GUID plus the old and new key IDs:

    stella key rotate <anchorId> <oldKeyId> <newKeyId> --overlap-days 7 \
  --algorithm Ed25519 --public-key <new-key.pem>

    --overlap-days defaults to 30; --algorithm accepts Ed25519 (default), ES256, ES384, or RS256. This currently fails closed (“requires a configured key rotation service”) — until the API client is wired, perform the rotation via the Signer API directly (step 2 below).

  2. Production path — add a new key to the trust anchor (Signer API):

    # Requires scope signer:rotate (policy Signer.KeyManagement)
POST /api/v1/anchors/{anchorId}/keys
{ "keyId": "...", "publicKey": "<PEM>", "algorithm": "ES256", "expiresAt": "<iso8601>" }

# Then revoke the expired key (it remains valid for proofs signed before revocation):
POST /api/v1/anchors/{anchorId}/keys/{keyId}/revoke
{ "reason": "expired - rotated", "effectiveAt": "<iso8601>" }

    Allowed algorithm strings (Signer KeyRotationOptions.AllowedAlgorithms): ES256, ES384, ES512, RS256, RS384, RS512, PS256, PS384, PS512, ED25519/EdDSA, SM2, GOST12-256, GOST12-512, DILITHIUM3, FALCON512 (plus deprecated RSA-2048, SHA1-RSA).

  3. Verify signing works by creating and verifying a test attestation (see Verification).

Root cause fix

Rotate to a fresh signing key:

  1. Add the new key to the trust anchor via the Signer API (POST /api/v1/anchors/{anchorId}/keys) with the desired algorithm and expiresAt. (Once the CLI client is wired, the equivalent is stella key add <anchorId> <keyId> --algorithm <alg> --public-key <pem>; it fails closed today.)

  2. Keep an overlap window so verifiers accept both old and new keys during transition. Until the expired key is revoked, both stay in the anchor’s AllowedKeyIds; the wired CLI helper (currently fail-closed) is:

    stella key rotate <anchorId> <oldKeyId> <newKeyId> --overlap-days 14

  3. Revoke the expired key once the overlap window closes (POST /api/v1/anchors/{anchorId}/keys/{keyId}/revoke). Revocation is audited (KeyAuditLogEntity, operation Revoke).

Set up expiration monitoring:

Monitoring today is not a single CLI command. Use one (or both) of:

Wire either signal into your existing notification routing (stella notify rules ...).

NOT IMPLEMENTED — stella notify channels add: the stella notify channels group exposes only list, show, and test subcommands (CommandFactory.cs) — there is no add/create subcommand, and no --event / --threshold-days options. The stella notify channels add --type email --event key.expiring --threshold-days 30 line that the SigningKeyExpirationCheck emits as remediation text (and that earlier drafts repeated) does not resolve to a real command. Channel CRUD is managed elsewhere (Console / Notify API); the CLI only lists/shows/tests channels and manages routing rules.

ROADMAP / NOT IMPLEMENTED (other): automatic-rotation config keys referenced in earlier drafts (rotation.auto, rotation.before_expiry, rotation.overlap_days, alerts.expiring_days, alerts.expiring_days_critical) and the commands stella keys generate, stella keys register, stella keys activate, stella keys config set, stella attest config set, stella attest reload, stella attest test-sign, stella attest retry, stella issuer keys publish do not exist in the current codebase. Expiry monitoring today is the Doctor check check.attestation.keymaterial (thresholds 30/7 days) plus the Signer warnings endpoint (default 60-day expiry warning). The Helm autoRotate snippet is aspirational.

Verification

# Re-run the Doctor key-material check (should no longer Fail/Warn):
stella doctor run --check check.attestation.keymaterial

# Show rotation status + warnings for the anchor (wired CLI; fail-closed until API client lands):
stella key status <anchorId>

# Verify an attestation chain attached to an OCI artifact (CLI prints sample output — see note below):
stella verify attestation --image <image-ref>

# Confirm a key's validity at a signing time via the Signer API:
GET /api/v1/anchors/{anchorId}/keys/{keyId}/validity?signedAt=<iso8601>

CLI shape (corrected) + demo caveat: stella verify attestation (in Commands/VerifyCommandGroup.cs → BuildVerifyAttestationCommand) takes a required --image/-i option — not a positional argument — plus optional --predicate-type/-t, --policy/-p, --output/-o, and --strict. As shipped it prints hardcoded sample output (verified = true with placeholder signers); it does not yet perform real chain verification, so do not treat its output as authoritative. The Signer validity API (above) is the trustworthy check today.

NOT IMPLEMENTED: stella attest create --type test ..., stella attest test-sign, and stella verify attestation --last are not part of the CLI.

Prevention