Runbook — Verification root-key rotation & re-enveloping

Audience: Platform / Crypto operators. Applies to: the signature-verification root keys, per regional crypto profile (world | fips | gost | sm | kcmvp | eidas), for three material types:

Related ADRs: ADR-028 (this procedure + the four binding decisions D1–D4), ADR-007 (KEK rotation — distinct concern; the AEAD that seals VexHub secrets), ADR-019 (VEX verification as a mandatory ingest gate). Related runbooks: kek-rotation.md, credential-storage-algorithm-switch.md, operator-verify.md.

What this rotates vs. what it does NOT. This runbook rotates the keys used to verify existing signed material. It is not the master-KEK rotation (kek-rotation.md) — though the two interact: VexHub’s HMAC trust-root secrets are sealed by the master KEK, so if you are doing both, re-seal VexHub secrets after the new signing root is active (ADR-028 Decisions & Risks).

Required scopes. Read verbs (list roots / get / re-envelope status) require the crypto:read claim (policy platform.crypto.read / CryptoProviderRead). Consequential mutations (introduce / invalidate) and the interactive progress-ingest fallback require the crypto:profile:admin claim (policy platform.crypto.profile.admin / CryptoProfileAdmin) — the same region-changing scope used for compliance-profile mutation. ops.admin is the operator escape hatch for both. Scope catalog: src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs; policy constants: src/Platform/StellaOps.Platform.WebService/Constants/PlatformPolicies.cs.

The model (read this first)

A verification root moves through a small lifecycle (mirrors the IssuerDirectory operator-key lifecycle):

                introduce              re-envelope            invalidate
  (none) ────────────────▶ active ──────────────────▶ active ───────────────▶ active
                            │                                                   ▲
   prior active ────────────┘                                                   │
        │ demoted to                                                            │
        ▼                                                                       │
     retiring  ──────────── (dual-trust grace: BOTH verify) ───────────────────┤
                                                                                │
                                                          ┌─ archived (D2: verify-only) ─┘  (RetainArchived=true)
                                                          └─ revoked  (D3: old material FAILS) (RetainArchived=false)

Three phases, applied per profile + per material type:

  1. Introduce (dual-trust) — add the NEW root; the current active root becomes retiring. Both verify during the grace window. Consequences-gated.
  2. Re-envelope — re-sign existing material under the new root. Resumable, offline-capable, idempotent.
  3. Invalidate (consequences-gated) — remove the OLD root (archived under the material-specific semantics below, or revoked). For Pack roots, either state leaves live trust and old signatures fail by design.

Fail-closed invariant (do not break it): a botched rotation may only ever degrade to “verification FAILS” (safe), never “verification skipped”. The boot-time verification-settings pull and every runtime guard stay fail-closed. For a real Pack ledger, Platform requires exactly one active member and every active/retiring member’s valid RSA public PEM; corrupt JSON, missing material, duplicate key ids or normalized RSA keys, case-insensitive key-id collisions or lifecycle-key variants, or an active-scalar mismatch aborts the pull. No retiring member is silently omitted. Pack key ids must be in exact trimmed form, contain no control characters, and fit within 256 UTF-8 bytes. Platform rejects a violating introduce before it writes anything and refuses to publish a retained violating row. Do not truncate or relabel historical ids; recover the exact retained state through an owner-approved audited repair.

PacksRegistry verification-settings wire contract

The existing GET /platform/verification-settings/packsregistry?profile=<profile> route returns three flat values (there is no rotation-specific endpoint):

{
  "PacksRegistry:Verification:PublicKeyPem": "<active PEM, unchanged compatibility scalar>",
  "PacksRegistry:Verification:TrustSetJson": "{\"schema\":\"stellaops.packsregistry.verification-trust-set/v1\",\"regionProfile\":\"world\",\"roots\":[{\"keyId\":\"sha256-new\",\"status\":\"active\",\"publicKeyPem\":\"...\"},{\"keyId\":\"sha256-old\",\"status\":\"retiring\",\"publicKeyPem\":\"...\"}]}",
  "PacksRegistry:Verification:TrustEpoch": "sha256:<lowercase hex>"
}

TrustSetJson is canonical JSON with fixed property order. Roots use lowercase wire statuses and are sorted ordinally by keyId, then status; PEM text is not trimmed or newline-normalized. The epoch hashes UTF-8 without BOM over <profile> + "\n" + <canonical-json>. Live trust is active + every retiring root. Archived Pack roots are audit-only and revoked roots are never included. Exactly one RSA PUBLIC KEY or RSA PUBLIC KEY PEM block plus surrounding whitespace is accepted. Never send a private key to Platform: private-key or mixed content is rejected before any introduce write and is never eligible for the scalar-only compatibility response. For collision checks, Platform imports either public PEM form, exports canonical DER SubjectPublicKeyInfo, and hashes it with SHA-256. Thus two live roots cannot publish the same RSA key under different ids or alternate PUBLIC KEY / RSA PUBLIC KEY encodings.

Every Pack verification-settings pull reads one coherent snapshot directly from the owning store; it does not use the replica-local general settings cache. Once an invalidation transaction commits, the next pull through any Platform replica therefore excludes the old root even if that replica’s generic cache is stale. Case-insensitive duplicate scalar, Pack-ledger, or Pack-progress keys fail closed before publication or mutation and require operator recovery. A unique row with noncanonical casing is not duplicated: conditional updates/deletes retain its exact physical spelling, while a genuinely absent row is created canonically. This keeps PostgreSQL and in-memory lifecycle behavior identical.

A scalar-only legacy install is returned as one synthetic active member with id legacy-pem-sha256-<sha256-of-the-exact-UTF8-PEM-text>. This name is a digest of the stored text, not a claim about the DER key fingerprint. On the first lifecycle introduce, Platform persists that same synthetic member as retiring before replacing the scalar, so a newly started consumer still receives old + new trust. A legacy active ledger row missing publicMaterial is backfilled from the scalar before demotion; if recovery is impossible, introduce performs no write and tells the operator to repair the ledger/scalar.

An older archived/revoked Pack row may predate public-material retention. Such a row blocks every later introduce because Platform cannot reconstruct or compare its RSA identity safely. Do not delete it, guess its key, or use generic environment-setting PUT/DELETE (those routes intentionally reject lifecycle rows). Recover the exact original public PEM from custody/audit evidence and request an owner-approved, audited ledger repair. If the exact PEM is unavailable, leave rotation blocked and escalate; invalidation does not make an unknown key safe to reuse.

Pack introduce is serialized per region profile and commits the retiring record, new active record, active scalar, and initial re-envelope status atomically. If two operators or replicas introduce different keys from the same prior state, exactly one commits; the other receives HTTP 409 verification_root_state_conflict and must re-read before deciding whether to start another rotation. The same 409 applies to a raced invalidate or Pack progress report; ordinary validation remains 400. If an operator loses the response after commit, retrying the exact same key id and PEM is safe and returns the already-committed active transition without duplicating audit or resetting re-envelope progress. Never change the PEM while reusing that key id: Platform rejects different material. Key ids are case-insensitively unique, so a spelling that differs from an existing id only by case is rejected before any write rather than treated as a new root. A different key id is not an escape hatch: introduce compares the normalized RSA SPKI fingerprint against every retained Pack ledger row, including retiring, archived, and revoked history, and rejects the write when the physical key was seen before. Only the exact active id + exact PEM retry is idempotent; after invalidation generate a genuinely new RSA key rather than relabeling the old one. The id itself must also be exact trimmed, control-free, and at most 256 UTF-8 bytes; the limit is a byte limit, so check multibyte ids before submitting the rotation.

PacksRegistry durable tenant/profile signature floor

The Platform trust publication is process input; the PacksRegistry floor is the durable storage admission boundary. For each tenant being re-enveloped, the first sweep call records (tenant_id, region_profile, active_key_id) under the global admission lock and records the observed trust epoch. The key id, not the epoch, is the monotonic security identity. A later publication may legitimately report the same active key with a different epoch (for example after another root is revoked); PacksRegistry updates the current epoch and appends the distinct observation to pack_signature_floor_epochs. Never compare epochs lexically, treat them as a generation counter, or delete old epoch facts.

For the next A-to-B rotation, the coherent active+retiring publication must identify the currently floored A as retiring before B may advance the floor. Do not supply guessed or historical key ids as the superseded set. Once B is committed, the floor survives every process restart and is never cleared after sweep completion. A stale host that still believes A is active receives one of these typed conflicts before publication:

The profile predicate is exact: a world floor does not couple fips, and vice versa. Pack-id ordering is never part of admission; a stale replica cannot publish a newly discovered lower-id pack after another replica advances the floor. Same-marker seed restart validation may use the live dual-trust set during grace, but every changed or new generation row must be accepted by the exact floored active key. The in-flight-claim fixed-point check remains tenant-wide because each profile sweep rewrites the same catalog; wait for claims from every profile before advancing a sweep.

Before declaring a tenant sweep complete, confirm both the fixed point and the durable facts:

SELECT tenant_id, region_profile, active_key_id, trust_epoch, advanced_at
FROM packs.pack_signature_floors
WHERE tenant_id = '<tenant>' AND region_profile = '<profile>';

SELECT active_key_id, trust_epoch, observed_at
FROM packs.pack_signature_floor_epochs
WHERE tenant_id = '<tenant>' AND region_profile = '<profile>'
ORDER BY observed_at, active_key_id, trust_epoch;

SELECT pack_id, catalog_state, artifact_origin, upload_operation_id,
       upload_verification_profile, upload_accepting_key_id, upload_trust_epoch
FROM packs.packs
WHERE tenant_id = '<tenant>'
  AND (catalog_state = 'staging'
       OR (artifact_origin = 'seed' AND upload_operation_id IS NOT NULL));

Expected: exactly one current floor for the tenant/profile, append-only epoch observations, and no in-flight rows. A missing floor after a completed sweep is not an invitation to recreate it by guessing; stop and recover from PostgreSQL/audit evidence.

Do not edit Pack scalar, ledger, or re-envelope rows through generic environment settings. Generic PUT/DELETE reserves those families for the lifecycle. The one bootstrap exception is an exact scoped key for a known profile (Verification:<profile>:PacksRegistry:PublicKeyPem) while no Pack ledger exists; it accepts only public RSA PEM, an exact retry is idempotent, and changing the PEM is rejected. Existing unscoped legacy rows remain readable but are not generically mutable; run lifecycle introduce to capture them safely. A blocked generic PUT or DELETE returns HTTP 409 with stable code verification_setting_lifecycle_owned. If the strict bootstrap loses its conditional-state race it instead returns 409 verification_root_state_conflict; re-read the lifecycle state before retrying.

Before you start

  1. Confirm the active region profile. Rotation is per-profile; rotate the profile your install actually verifies under. Check it via the Console Cryptography page (/setup/system/cryptography) or PUT/GET /api/v1/admin/crypto-providers/compliance-profile (tenant tenant_compliance_profile).

  2. List the current roots for the profile + material type you are rotating:

    stella crypto verification-key list --profile world --material-type pack-public-key

    or via the API (read scope platform.crypto.read):

    curl -sH "Authorization: Bearer $TOKEN" \
  https://stella-ops.local/platform/verification-keys/world | jq

    You should see exactly one active root and no pending/retiring leftovers from a prior incomplete rotation. If a prior rotation is mid-flight (a retiring root still present, or a non-zero residual on its re-envelope status), finish or invalidate that first — do not stack rotations. Platform enforces this: a distinct Pack introduce writes nothing while a retiring root exists, while an exact active-id/PEM retry remains safe.

    Direct lifecycle reads also fail closed when a profile exceeds 256 retained root rows, a lifecycle value exceeds 64 KiB UTF-8, or the profile snapshot exceeds 4 MiB UTF-8. These are recovery bounds, not permission to delete or truncate audit history; escalate for an owner-approved exact repair.

    The complete JSON response consumed at startup has its own 64 KiB ceiling. Platform validates the exact projected outer dictionary before introduce or legacy-bootstrap writes, so individually valid PEM rows that would overflow the active+retiring response are rejected without mutation. A retained over-limit response is a fail-closed recovery incident.

  3. Have the NEW key material ready.

    • pack-public-key: the new RSA public key PEM (for the active set) and the matching private key PEM (offline re-signing only — never submit or persist it in Platform). Introduce accepts only PUBLIC KEY / RSA PUBLIC KEY.
    • vex-hmac-trust-root: the new HMAC-SHA256 secret (in a file; it is sealed, never echoed) and its sha256- fingerprint.
    • signing-root: the new signing key enrolled in the regional crypto registry (KeyId), and its public PEM.

Fingerprint form. Key-ids / fingerprints MUST use the delimiter-safe sha256- form, never sha256:. The colon form is parsed by the .NET configuration binder as an extra nesting level and the trust root is silently dropped (docs/architecture/runtime-configuration-validation.md §6). verification-key introduce rejects sha256: up front.

Phase 1 — Introduce the new root (begins dual-trust grace)

This is consequences-gated. There is no silent root change from any surface.

CLI

stella crypto verification-key introduce \
  --profile world \
  --material-type pack-public-key \
  --key-id sha256-<new-fingerprint> \
  --public-material-file ./new-pack-public.pem \
  --grace-deadline 2026-07-01T00:00:00Z \
  --confirm \
  --i-understand-old-material-will-be-invalidated

Without both --confirm AND --i-understand-old-material-will-be-invalidated, the command refuses, prints the exact typed consequences phrase, and exits non-zero (13) — no mutation.

For a VexHub HMAC trust root the secret is sealed, never inlined:

stella crypto verification-key introduce \
  --profile world --material-type vex-hmac-trust-root \
  --key-id sha256-<new-fingerprint> \
  --secret-file ./new-vex-hmac.secret \
  --confirm --i-understand-old-material-will-be-invalidated

For a signing root pass the public PEM via --public-material-file (the private key lives in the regional crypto registry, not here).

API

curl -sX POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  https://stella-ops.local/platform/verification-keys/world/introduce \
  -d '{
        "keyId": "sha256-<new-fingerprint>",
        "materialType": "pack-public-key",
        "publicMaterial": "-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----",
        "graceDeadlineUtc": "2026-07-01T00:00:00Z",
        "acknowledgement": "<typed phrase>"
      }'

If the acknowledgement is missing or wrong, the endpoint returns 422 consequences_acknowledgement_required with the exact requiredPhrase you must echo. (For VexHub HMAC, send secretMaterial instead of publicMaterial; the secret is sealed and never echoed back.)

What happens on introduce:

Verify the grace state:

stella crypto verification-key list --profile world --material-type pack-public-key
# expect: one active (new key), one retiring (old key) with the grace deadline.

At this point both old and new material verify. Do not invalidate yet.

Phase 2 — Re-envelope existing material under the new root

Re-enveloping re-signs/re-seals the existing corpus so it verifies under the new root. It is resumable, idempotent, and offline-capable (no external services — see the air-gap section). Run it per material type.

Packs

stella crypto verification-key reenvelope \
  --profile world \
  --new-key-id sha256-<new-fingerprint> \
  --new-public-key-file ./new-pack-public.pem \
  --new-private-key-file ./new-pack-private.pem

This launches the bulk job (POST /api/v1/packs/re-envelope, PackReEnvelopeJob): it iterates the pack corpus, re-signs each pack under the new key via RotateSignatureAsync (re-verifies + emits signature.rotated), and reports total / re-enveloped / failed / skipped progress back to the foundation status endpoint. A pack whose current signature already verifies under the new key is skipped (idempotent), so a second run does no work. Progress persists to a durable per-(tenant, profile, newKeyId) checkpoint, so an interrupted run resumes without re-processing converted packs.

DSSE attestations / evidence bundles (signing root)

The CapsuleReEnvelopeJob (EvidenceLocker) adds a new-keyId signature to each capsule’s DSSE envelope (multi-sig — the old signature is preserved so already-distributed bundles still verify; the payload is never mutated), re-verifies the merged envelope under the new key before persisting (a bad signature is a failure, never persisted = fail-closed), is idempotent, resumable, and offline.

D2 — immutable / transparency-anchored material is NOT re-signed. A capsule flagged immutable (e.g. a Rekor-anchored entry) is recorded Immutable / skipped, not failed. It keeps verifying via the archived root you retain at invalidation. Do not try to re-issue it.

VEX

Re-sign / re-ingest the VEX statements under the new trust root (the publisher re-signs with the new HMAC secret; VexHub then verifies under the new fingerprint). The new fingerprint is already served alongside the old during grace.

Monitor progress

stella crypto verification-key reenvelope-status \
  --profile world --material-type pack-public-key

or:

curl -sH "Authorization: Bearer $TOKEN" \
  https://stella-ops.local/platform/verification-keys/world/pack-public-key/reenvelope-status | jq
# { total, reEnveloped, failed, skipped, completed, residual }

The same progress is rendered in the Console Cryptography page’s “Root rotation lifecycle” view (progress bar + residual banner). residual = total − reEnveloped − skipped is the count of not-yet-re-enveloped artifacts.

Long-sweep discipline. Re-enveloping a large corpus is a long, resumable job — monitor it; never run it unmonitored (repo §long-sweep rule). It is safe to interrupt and resume.

Re-envelope is complete when completed: true and residual: 0.

Phase 3 — Invalidate the old root (consequences-gated)

Once re-enveloping is complete (or you explicitly accept the residual — D4), remove the OLD root. From this point, old signatures fail verification by design.

Choose archive vs. revoke

CLI

stella crypto verification-key invalidate \
  --profile world \
  --material-type pack-public-key \
  --key-id sha256-<OLD-fingerprint> \
  --confirm \
  --i-understand-old-material-will-be-invalidated

(Add --retain-archived to archive instead of revoke.) Without both consequence flags, the command refuses, prints the consequences, and exits non-zero.

API

curl -sX POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  https://stella-ops.local/platform/verification-keys/world/invalidate \
  -d '{
        "keyId": "sha256-<OLD-fingerprint>",
        "materialType": "pack-public-key",
        "retainArchived": false,
        "acknowledgement": "<typed phrase>"
      }'

A wrong/missing acknowledgement returns 422 with the requiredPhrase.

What happens on invalidate:

Post-rotation verification

  1. List shows the new state:

    stella crypto verification-key list --profile world --material-type pack-public-key
# expect: one active (new key); the old key archived OR absent (revoked).

  2. New material verifies; old material fails. This is the security outcome the e2e test proves (VerificationKeyRotationE2ETests, src/Platform/__Tests/StellaOps.Platform.WebService.Tests/): post-invalidation, material signed under the new root verifies and material signed only under the revoked root fails — for packs (RSA), signing roots (DSSE), and VEX (HMAC).

    For packs, pull GET /platform/verification-settings/packsregistry?profile=world from a fresh process. During grace, parse PacksRegistry:Verification:TrustSetJson and expect one active plus every retiring root. After revoke, expect only the new active root and a changed PacksRegistry:Verification:TrustEpoch.

  3. Immutable material still verifies (D2). If you archived (not revoked), a transparency-anchored DSSE entry signed under the old key still verifies via the archived JWKS entry — no re-issue.

  4. Signing-root JWKS reflects the change. EvidenceLocker/Attestor pull the set at GET /platform/verification-keys/{profile}/signing-root/jwks-set (30s cache, fail-closed to empty on error). After a revoke the old KeyId is gone from the active set; after an archive it remains with status: archived.

  5. Audit trail. The timeline carries verification.key.{introduced, reenvelope_started, reenvelope_completed, invalidated} with correlation ids and non-secret counters only.

Consequences & recovery

What invalidation actually breaks

Revoking a root makes every artifact signed only under it stop verifying. That is the point — for a retired/compromised key it is the desired outcome. The danger is invalidating before re-enveloping is complete: any not-yet-re-enveloped artifact then fails verification. D4 lets you proceed past the grace deadline with a residual, but the residual artifacts will fail until re-enveloped.

“I invalidated too early — material that should verify now fails.”

“A service won’t start after rotation.”

That is the fail-closed guard working as designed: the boot-time pull found no active root for the current profile (e.g. you revoked the sole active root with no replacement). Fix forward — introduce a valid active root for that profile, then restart. Never disable the guard.

“I need to abort an in-flight rotation.”

Before invalidating the retiring root, re-envelope is idempotent and both keys still verify, so the safest pause is to leave the dual-trust state intact while you choose a fix-forward path. The lifecycle does not automatically promote a retiring root. Invalidating the new active key clears the active scalar and makes Pack publication fail closed; it does not restore the old root or the pre-rotation state. That zero-active state does not waive the one-in-flight-rotation gate: while the old root remains retiring, a distinct introduce writes nothing. Finish the consequences review and invalidate that retiring root, or recover an operator-reviewed PostgreSQL snapshot and reconcile artifacts already re-enveloped after it. Only a zero-active, scalar-absent state with no retiring root may accept one cryptographically distinct replacement; archived/revoked history remains preserved. Never repair these rows through generic environment-settings PUT/DELETE.

Air-gap / offline operation

The entire rotation cycle is air-gap-safe and requires no external services:

The procedure on an air-gap install is identical to the steps above; only ensure the new key material is staged on the deployment’s internal media before you begin.