ADR-028: Verification root-key rotation and signed-material re-enveloping

Status

Accepted — 2026-06-10 (operator-approved; the four open questions below are resolved)

Context

Recent work (commits 7a4f7921e2 backend, 7446116190 Console UI) made signature-verification material admin-settable per regional crypto profile (world|fips|gost|sm|kcmvp|eidas) via Platform EnvironmentSettings, under the reserved namespace Verification:<profile>:<service>:..., consumed by services at boot through PlatformVerificationSettingsConfigurationSource (a fail-closed boot-time pull from GET /platform/verification-settings/{service}). Today the in-scope material is:

Those keys are currently seeded with test values for the world profile. Production needs:

  1. Operators to set real keys from both UI and CLI.
  2. A change to a verification root key to require an explicit “this has consequences” acknowledgment.
  3. A defined procedure to re-envelope existing signed material under the next key and then invalidate the old material.

Removing a verification root key makes every artifact signed under it stop verifying. A naive hard cutover would instantly break verification of all in-flight signed material (packs, DSSE attestations, VEX envelopes, evidence bundles). We must not do that.

Existing primitives we build on (verified in src/):

Decision

Adopt a phased, dual-trust rotation with explicit re-enveloping and a consequences-gated invalidation — never a hard cutover. Three phases, applied per region profile and per material type:

  1. Introduce (dual-trust). The operator adds the NEW verification root alongside the current one; both verify during a grace window. New material is signed with the new key; old material continues to verify under the old root. The new root is stored with status pending/active; the old becomes retiring. A warning is shown that a rotation has begun. No old material is invalidated yet.

  2. Re-envelope. A resumable, offline-capable re-enveloping job re-signs existing signed material under the new key:

    • Packs → bulk RotateSignatureAsync with the new key (iterates the pack corpus; idempotent; emits signature.rotated).
    • DSSE attestations / evidence bundles → add a new-keyId signature to each envelope (multi-sig) — or re-issue where re-signing is impossible.
    • VEX → re-sign / re-ingest under the new trust root. Progress is tracked per material type (total / re-enveloped / failed / skipped), is resumable, and runs without external services (air-gap-safe).

  3. Invalidate (consequences-gated). Once re-enveloping is complete (or the operator explicitly accepts the residual), the operator removes the OLD root. From this point, old signatures fail verification by design. This step requires an explicit typed consequences acknowledgment (UI confirmation dialog requiring the operator to type a confirmation phrase; CLI --confirm plus an --i-understand-old-material-will-be-invalidated flag) and is fully audited.

This applies to all roots — the env_settings verification roots and the regional crypto signing root (UseCryptoRegistry + KeyId) that attestor/evidence use for capsule/verdict signing. Signing-root rotation publishes both the old and new public keys in the JWKS during the grace window, re-envelopes (re-signs/re-seals) material under the new KeyId, then drops the old KeyId from the active JWKS — retaining the old public key as an archived root (see below).

Cross-cutting rules:

Consequences

Alternatives considered

Resolved decisions (2026-06-10, operator)

  1. Secret storage — secrets are never exposed in env_settings. Non-secret verification material (public keys, fingerprints) lives in env_settings; secret material (VexHub HMAC trust roots) is sealed in the platform encrypted credential store (ConnectorCredentials AEAD / ICredentialStore, regional-crypto-backed, ConnectorCredentialReSealStore for rotation). Vault is NOT required (optional integration only).
  2. Immutable / transparency-anchored material — verify it via a retained archivedroot (verify-only). Do NOT re-issue.
  3. Scope = all roots — rotate both the env_settings verification roots and the regional crypto signing root (UseCryptoRegistry/KeyId, JWKS-published) for attestor/evidence.
  4. Grace deadline with re-envelope incomplete — warn-and-allow: invalidation proceeds with a residual-count warning; it is not blocked.

Pack trust publication addendum (2026-08-20)

The original single-valued materialization Verification:<profile>:PacksRegistry:PublicKeyPem is retained for compatibility, but it cannot by itself preserve dual trust across a process restart. The following rules complete the producer side of the Pack rotation contract without adding an endpoint:

  1. Every PackPublicKey ledger record persists its exact, non-secret publicMaterial, just as a SigningRoot record does. Before a legacy active Pack record that lacks this field is demoted, Platform captures the current materialized scalar into that record. If the scalar is absent, inconsistent, or not valid RSA public-key PEM, introduce fails before its first store write and requires operator recovery. Exactly one PUBLIC KEY or RSA PUBLIC KEY PEM block (plus surrounding whitespace) is accepted; private-key or mixed content is rejected before persistence so secret material cannot enter this non-secret ledger/response surface.
  2. A scalar-only legacy installation (no Pack ledger rows) remains readable. Platform publishes a synthetic active member named legacy-pem-sha256-<sha256-of-the-exact-UTF8-PEM-text>. On the first lifecycle introduce, that same identity and exact PEM are persisted as the retiring root before the new scalar replaces it; the new record’s supersedesKeyId points to the synthetic identity.
  3. The existing GET /platform/verification-settings/{service} response for PacksRegistry carries all three flat configuration values:
    • PacksRegistry:Verification:PublicKeyPem — unchanged active scalar;
    • PacksRegistry:Verification:TrustSetJson — canonical JSON {"schema":"stellaops.packsregistry.verification-trust-set/v1","regionProfile":"<profile>","roots":[{"keyId":"...","status":"active|retiring","publicKeyPem":"..."}]};
    • PacksRegistry:Verification:TrustEpoch — sha256: plus lowercase SHA-256 of UTF-8 without a BOM over <profile> + "\n" + <canonical-json>.
  4. Canonical roots use lowercase wire statuses and are sorted by ordinal keyId, then status. Every Pack keyId is stored in exact trimmed form, contains no control characters, and is at most 256 UTF-8 bytes. Introduce rejects an invalid id before any write, and publication fails closed when retained history violates that contract; operators must recover the exact history rather than truncate or relabel it. A real ledger must have exactly one active root; every retiring root is included. Missing or malformed ledger JSON, missing/invalid PEM, duplicate key ids or normalized RSA keys, case-insensitive key-id collisions or lifecycle-key variants, or an ordinal mismatch between the active ledger PEM and active scalar fails closed. No real-ledger member is silently omitted. Pack publication reads one direct coherent store snapshot on every pull, bypassing the general replica-local settings cache, so the first pull through any replica after invalidation commit cannot re-publish the revoked member.
  5. Live Pack verification includes active + at most one retiring root only. A distinct introduce is rejected atomically while a prior root remains retiring; the operator must finish re-enveloping and invalidate it before starting another rotation. An exact retry of the already committed active id + PEM remains idempotent. Pack artifacts are mutable and re-enveloped, so an archived Pack root is ledger/audit history, not live trust. Signing-root archived semantics remain unchanged: immutable/transparency-anchored material may continue to verify through archived signing-root JWKS members. Revoked roots are never published.
  6. Pack introduce is one conditional store mutation, not a sequence of independently visible setting writes. Platform captures an expected-state token over the scoped and legacy-unscoped active scalars, every Pack ledger row, and Pack re-envelope progress. PostgreSQL serializes by a stable profile-scoped transaction advisory lock, directly re-reads and compares that state after taking the lock, and commits retiring metadata, new active metadata, the scoped scalar, and the initial re-envelope status in one transaction. When a unique pre-existing lifecycle row uses noncanonical casing, updates and deletes target that exact physical key; only an absent row is created with canonical spelling. The in-memory implementation mirrors this under one gate. A changed token raises a typed conflict and writes nothing, so concurrent different-key introduces cannot create two active roots or choose the scalar by last writer. Retrying the exact active key id and ordinal-equal PEM after a lost response returns the committed result without another transition; the same key id with different material is rejected. A different key id is also rejected when its RSA key matches any retained Pack ledger member after canonical ExportSubjectPublicKeyInfo normalization and SHA-256, including alternate PKCS#1/SPKI PEM encodings and archived/revoked history. Lifecycle HTTP mutations surface the typed race as 409 with stable code verification_root_state_conflict; other validation remains 400.
  7. Introduce, invalidate, Pack re-envelope progress, and legacy bootstrap are the only application writers of those rows and all use the same conditional mutation. Generic environment-setting Set, batch Set, and Delete reject the unscoped/scoped Pack scalar, Pack metadata, and Pack re-envelope families. The existing admin PUT retains one narrow bootstrap: an exact Verification:<known-profile>:PacksRegistry:PublicKeyPem may establish a strictly validated scalar only while the Pack ledger is empty. Exact material retry is idempotent; replacement, unknown-profile/unscoped writes, metadata/progress writes, and every generic delete are rejected. Rejected generic lifecycle-owned PUT/DELETE calls return 409 verification_setting_lifecycle_owned; a strict-bootstrap conditional race returns 409 verification_root_state_conflict. Pre-existing unscoped legacy rows remain readable and are captured by lifecycle introduce.
  8. Direct Pack lifecycle snapshots are bounded before materialization: at most 256 retained root rows, at most 64 KiB UTF-8 per lifecycle value, and at most 4 MiB UTF-8 for the complete profile snapshot. PostgreSQL reads length columns first under sequential access and fetches at most one row beyond the row ceiling. Both supported mutation stores validate the complete projected state against the same bounds before their first write. Exceeding any bound fails publication and mutation closed for owner-approved recovery; no retained history is silently omitted or truncated.
  9. Invalidating the sole active Pack root deliberately leaves publication fail-closed; no retiring root is auto-promoted or rewritten. Recovery may introduce one cryptographically distinct new active root only when the real ledger has zero active members, no retiring member, and no scoped or unscoped active scalar row. Existing archived/revoked rows and their audit history remain unchanged and reserve their normalized RSA keys against relabeling/reintroduction. Any scalar-present or multi-active mismatch still requires operator recovery and writes nothing. A retained terminal row that predates publicMaterial retention also blocks introduce: its identity cannot be reconstructed safely, the generic settings API cannot repair it, and the operator must recover the exact original public PEM for an owner-approved audited ledger repair or keep rotation fail-closed.
  10. Platform and the startup consumer share one 64 KiB UTF-8 transport ceiling for the complete serialized PacksRegistry verification-settings response, not merely for each retained row. Introduce and legacy bootstrap serialize the exact projected outer dictionary with the endpoint JSON options before their first write; one byte over commits nothing. Publication independently applies the same check to retained state and fails closed for bounded operator recovery.

References