Runbook — Verification root-key rotation & re-enveloping

Audience: Platform / Crypto operators. Applies to: the signature-verification root keys, per regional crypto profile (world | fips | gost | sm | kcmvp | eidas), for three material types:

Related ADRs: ADR-028 (this procedure + the four binding decisions D1–D4), ADR-007 (KEK rotation — distinct concern; the AEAD that seals VexHub secrets), ADR-019 (VEX verification as a mandatory ingest gate). Related runbooks: kek-rotation.md, credential-storage-algorithm-switch.md, operator-verify.md.

What this rotates vs. what it does NOT. This runbook rotates the keys used to verify existing signed material. It is not the master-KEK rotation (kek-rotation.md) — though the two interact: VexHub’s HMAC trust-root secrets are sealed by the master KEK, so if you are doing both, re-seal VexHub secrets after the new signing root is active (ADR-028 Decisions & Risks).

Required scopes. Read verbs (list roots / get / re-envelope status) require platform.crypto.read (policy CryptoProviderRead). Consequential mutations (introduce / invalidate) require platform.crypto.profile.admin (policy CryptoProfileAdmin) — the same region-changing scope used for compliance-profile mutation. ops.admin is the operator escape hatch for both. Scope catalog: src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs; policy constants: src/Platform/StellaOps.Platform.WebService/Constants/PlatformPolicies.cs.

The model (read this first)

A verification root moves through a small lifecycle (mirrors the IssuerDirectory operator-key lifecycle):

                introduce              re-envelope            invalidate
  (none) ────────────────▶ active ──────────────────▶ active ───────────────▶ active
                            │                                                   ▲
   prior active ────────────┘                                                   │
        │ demoted to                                                            │
        ▼                                                                       │
     retiring  ──────────── (dual-trust grace: BOTH verify) ───────────────────┤
                                                                                │
                                                          ┌─ archived (D2: verify-only) ─┘  (RetainArchived=true)
                                                          └─ revoked  (D3: old material FAILS) (RetainArchived=false)

Three phases, applied per profile + per material type:

  1. Introduce (dual-trust) — add the NEW root; the current active root becomes retiring. Both verify during the grace window. Consequences-gated.
  2. Re-envelope — re-sign existing material under the new root. Resumable, offline-capable, idempotent.
  3. Invalidate (consequences-gated) — remove the OLD root (archived verify-only, or revoked). After this, old signatures fail by design.

Fail-closed invariant (do not break it): a botched rotation may only ever degrade to “verification FAILS” (safe), never “verification skipped”. The boot-time verification-settings pull and every runtime guard stay fail-closed — a missing/empty active root for the current profile still aborts service startup.

Before you start

  1. Confirm the active region profile. Rotation is per-profile; rotate the profile your install actually verifies under. Check it via the Console Cryptography page (/setup/system/cryptography) or PUT/GET /api/v1/admin/crypto-providers/compliance-profile (tenant tenant_compliance_profile).

  2. List the current roots for the profile + material type you are rotating:

    stella crypto verification-key list --profile world --material-type pack-public-key

    or via the API (read scope platform.crypto.read):

    curl -sH "Authorization: Bearer $TOKEN" \
  https://stella-ops.local/platform/verification-keys/world | jq

    You should see exactly one active root and no pending/retiring leftovers from a prior incomplete rotation. If a prior rotation is mid-flight (a retiring root still present, or a non-zero residual on its re-envelope status), finish or invalidate that first — do not stack rotations.

  3. Have the NEW key material ready.

    • pack-public-key: the new RSA public key PEM (for the active set) and the matching private key PEM (offline re-signing only — never persisted).
    • vex-hmac-trust-root: the new HMAC-SHA256 secret (in a file; it is sealed, never echoed) and its sha256- fingerprint.
    • signing-root: the new signing key enrolled in the regional crypto registry (KeyId), and its public PEM.

Fingerprint form. Key-ids / fingerprints MUST use the delimiter-safe sha256- form, never sha256:. The colon form is parsed by the .NET configuration binder as an extra nesting level and the trust root is silently dropped (docs/architecture/runtime-configuration-validation.md §6). verification-key introduce rejects sha256: up front.

Phase 1 — Introduce the new root (begins dual-trust grace)

This is consequences-gated. There is no silent root change from any surface.

CLI

stella crypto verification-key introduce \
  --profile world \
  --material-type pack-public-key \
  --key-id sha256-<new-fingerprint> \
  --public-material-file ./new-pack-public.pem \
  --grace-deadline 2026-07-01T00:00:00Z \
  --confirm \
  --i-understand-old-material-will-be-invalidated

Without both --confirm AND --i-understand-old-material-will-be-invalidated, the command refuses, prints the exact typed consequences phrase, and exits non-zero (13) — no mutation.

For a VexHub HMAC trust root the secret is sealed, never inlined:

stella crypto verification-key introduce \
  --profile world --material-type vex-hmac-trust-root \
  --key-id sha256-<new-fingerprint> \
  --secret-file ./new-vex-hmac.secret \
  --confirm --i-understand-old-material-will-be-invalidated

For a signing root pass the public PEM via --public-material-file (the private key lives in the regional crypto registry, not here).

API

curl -sX POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  https://stella-ops.local/platform/verification-keys/world/introduce \
  -d '{
        "keyId": "sha256-<new-fingerprint>",
        "materialType": "pack-public-key",
        "publicMaterial": "-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----",
        "graceDeadlineUtc": "2026-07-01T00:00:00Z",
        "acknowledgement": "<typed phrase>"
      }'

If the acknowledgement is missing or wrong, the endpoint returns 422 consequences_acknowledgement_required with the exact requiredPhrase you must echo. (For VexHub HMAC, send secretMaterial instead of publicMaterial; the secret is sealed and never echoed back.)

What happens on introduce:

Verify the grace state:

stella crypto verification-key list --profile world --material-type pack-public-key
# expect: one active (new key), one retiring (old key) with the grace deadline.

At this point both old and new material verify. Do not invalidate yet.

Phase 2 — Re-envelope existing material under the new root

Re-enveloping re-signs/re-seals the existing corpus so it verifies under the new root. It is resumable, idempotent, and offline-capable (no external services — see the air-gap section). Run it per material type.

Packs

stella crypto verification-key reenvelope \
  --profile world \
  --new-key-id sha256-<new-fingerprint> \
  --new-public-key-file ./new-pack-public.pem \
  --new-private-key-file ./new-pack-private.pem

This launches the bulk job (POST /api/v1/packs/re-envelope, PackReEnvelopeJob): it iterates the pack corpus, re-signs each pack under the new key via RotateSignatureAsync (re-verifies + emits signature.rotated), and reports total / re-enveloped / failed / skipped progress back to the foundation status endpoint. A pack whose current signature already verifies under the new key is skipped (idempotent), so a second run does no work. Progress persists to a durable per-(tenant, profile, newKeyId) checkpoint, so an interrupted run resumes without re-processing converted packs.

DSSE attestations / evidence bundles (signing root)

The CapsuleReEnvelopeJob (EvidenceLocker) adds a new-keyId signature to each capsule’s DSSE envelope (multi-sig — the old signature is preserved so already-distributed bundles still verify; the payload is never mutated), re-verifies the merged envelope under the new key before persisting (a bad signature is a failure, never persisted = fail-closed), is idempotent, resumable, and offline.

D2 — immutable / transparency-anchored material is NOT re-signed. A capsule flagged immutable (e.g. a Rekor-anchored entry) is recorded Immutable / skipped, not failed. It keeps verifying via the archived root you retain at invalidation. Do not try to re-issue it.

VEX

Re-sign / re-ingest the VEX statements under the new trust root (the publisher re-signs with the new HMAC secret; VexHub then verifies under the new fingerprint). The new fingerprint is already served alongside the old during grace.

Monitor progress

stella crypto verification-key reenvelope-status \
  --profile world --material-type pack-public-key

or:

curl -sH "Authorization: Bearer $TOKEN" \
  https://stella-ops.local/platform/verification-keys/world/pack-public-key/reenvelope-status | jq
# { total, reEnveloped, failed, skipped, completed, residual }

The same progress is rendered in the Console Cryptography page’s “Root rotation lifecycle” view (progress bar + residual banner). residual = total − reEnveloped − skipped is the count of not-yet-re-enveloped artifacts.

Long-sweep discipline. Re-enveloping a large corpus is a long, resumable job — monitor it; never run it unmonitored (repo §long-sweep rule). It is safe to interrupt and resume.

Re-envelope is complete when completed: true and residual: 0.

Phase 3 — Invalidate the old root (consequences-gated)

Once re-enveloping is complete (or you explicitly accept the residual — D4), remove the OLD root. From this point, old signatures fail verification by design.

Choose archive vs. revoke

CLI

stella crypto verification-key invalidate \
  --profile world \
  --material-type pack-public-key \
  --key-id sha256-<OLD-fingerprint> \
  --confirm \
  --i-understand-old-material-will-be-invalidated

(Add --retain-archived to archive instead of revoke.) Without both consequence flags, the command refuses, prints the consequences, and exits non-zero.

API

curl -sX POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  https://stella-ops.local/platform/verification-keys/world/invalidate \
  -d '{
        "keyId": "sha256-<OLD-fingerprint>",
        "materialType": "pack-public-key",
        "retainArchived": false,
        "acknowledgement": "<typed phrase>"
      }'

A wrong/missing acknowledgement returns 422 with the requiredPhrase.

What happens on invalidate:

Post-rotation verification

  1. List shows the new state:

    stella crypto verification-key list --profile world --material-type pack-public-key
# expect: one active (new key); the old key archived OR absent (revoked).

  2. New material verifies; old material fails. This is the security outcome the e2e test proves (VerificationKeyRotationE2ETests, src/Platform/__Tests/StellaOps.Platform.WebService.Tests/): post-invalidation, material signed under the new root verifies and material signed only under the revoked root fails — for packs (RSA), signing roots (DSSE), and VEX (HMAC).

  3. Immutable material still verifies (D2). If you archived (not revoked), a transparency-anchored DSSE entry signed under the old key still verifies via the archived JWKS entry — no re-issue.

  4. Signing-root JWKS reflects the change. EvidenceLocker/Attestor pull the set at GET /platform/verification-keys/{profile}/signing-root/jwks-set (30s cache, fail-closed to empty on error). After a revoke the old KeyId is gone from the active set; after an archive it remains with status: archived.

  5. Audit trail. The timeline carries verification.key.{introduced, reenvelope_started, reenvelope_completed, invalidated} with correlation ids and non-secret counters only.

Consequences & recovery

What invalidation actually breaks

Revoking a root makes every artifact signed only under it stop verifying. That is the point — for a retired/compromised key it is the desired outcome. The danger is invalidating before re-enveloping is complete: any not-yet-re-enveloped artifact then fails verification. D4 lets you proceed past the grace deadline with a residual, but the residual artifacts will fail until re-enveloped.

“I invalidated too early — material that should verify now fails.”

“A service won’t start after rotation.”

That is the fail-closed guard working as designed: the boot-time pull found no active root for the current profile (e.g. you revoked the sole active root with no replacement). Fix forward — introduce a valid active root for that profile, then restart. Never disable the guard.

“I need to abort an in-flight rotation.”

Before invalidation, a rotation is fully reversible: re-envelope is idempotent and the old root is still retiring (both keys verify). To abort, simply do not invalidate; if you want to discard the new root, invalidate the new key-id (revoke) — the old root remains active and you are back to the pre-rotation state.

Air-gap / offline operation

The entire rotation cycle is air-gap-safe and requires no external services:

The procedure on an air-gap install is identical to the steps above; only ensure the new key material is staged on the deployment’s internal media before you begin.