ADR-028: Verification root-key rotation and signed-material re-enveloping

Status

Accepted — 2026-06-10 (operator-approved; the four open questions below are resolved)

Context

Recent work (commits 7a4f7921e2 backend, 7446116190 Console UI) made signature-verification material admin-settable per regional crypto profile (world|fips|gost|sm|kcmvp|eidas) via Platform EnvironmentSettings, under the reserved namespace Verification:<profile>:<service>:..., consumed by services at boot through PlatformVerificationSettingsConfigurationSource (a fail-closed boot-time pull from GET /platform/verification-settings/{service}). Today the in-scope material is:

Those keys are currently seeded with test values for the world profile. Production needs:

  1. Operators to set real keys from both UI and CLI.
  2. A change to a verification root key to require an explicit “this has consequences” acknowledgment.
  3. A defined procedure to re-envelope existing signed material under the next key and then invalidate the old material.

Removing a verification root key makes every artifact signed under it stop verifying. A naive hard cutover would instantly break verification of all in-flight signed material (packs, DSSE attestations, VEX envelopes, evidence bundles). We must not do that.

Existing primitives we build on (verified in src/):

Decision

Adopt a phased, dual-trust rotation with explicit re-enveloping and a consequences-gated invalidation — never a hard cutover. Three phases, applied per region profile and per material type:

  1. Introduce (dual-trust). The operator adds the NEW verification root alongside the current one; both verify during a grace window. New material is signed with the new key; old material continues to verify under the old root. The new root is stored with status pending/active; the old becomes retiring. A warning is shown that a rotation has begun. No old material is invalidated yet.

  2. Re-envelope. A resumable, offline-capable re-enveloping job re-signs existing signed material under the new key:

    • Packs → bulk RotateSignatureAsync with the new key (iterates the pack corpus; idempotent; emits signature.rotated).
    • DSSE attestations / evidence bundles → add a new-keyId signature to each envelope (multi-sig) — or re-issue where re-signing is impossible.
    • VEX → re-sign / re-ingest under the new trust root. Progress is tracked per material type (total / re-enveloped / failed / skipped), is resumable, and runs without external services (air-gap-safe).

  3. Invalidate (consequences-gated). Once re-enveloping is complete (or the operator explicitly accepts the residual), the operator removes the OLD root. From this point, old signatures fail verification by design. This step requires an explicit typed consequences acknowledgment (UI confirmation dialog requiring the operator to type a confirmation phrase; CLI --confirm plus an --i-understand-old-material-will-be-invalidated flag) and is fully audited.

This applies to all roots — the env_settings verification roots and the regional crypto signing root (UseCryptoRegistry + KeyId) that attestor/evidence use for capsule/verdict signing. Signing-root rotation publishes both the old and new public keys in the JWKS during the grace window, re-envelopes (re-signs/re-seals) material under the new KeyId, then drops the old KeyId from the active JWKS — retaining the old public key as an archived root (see below).

Cross-cutting rules:

Consequences

Alternatives considered

Resolved decisions (2026-06-10, operator)

  1. Secret storage — secrets are never exposed in env_settings. Non-secret verification material (public keys, fingerprints) lives in env_settings; secret material (VexHub HMAC trust roots) is sealed in the platform encrypted credential store (ConnectorCredentials AEAD / ICredentialStore, regional-crypto-backed, ConnectorCredentialReSealStore for rotation). Vault is NOT required (optional integration only).
  2. Immutable / transparency-anchored material — verify it via a retained archivedroot (verify-only). Do NOT re-issue.
  3. Scope = all roots — rotate both the env_settings verification roots and the regional crypto signing root (UseCryptoRegistry/KeyId, JWKS-published) for attestor/evidence.
  4. Grace deadline with re-envelope incomplete — warn-and-allow: invalidation proceeds with a residual-count warning; it is not blocked.

Pack trust publication addendum (2026-08-20)

The original single-valued materialization Verification:<profile>:PacksRegistry:PublicKeyPem is retained for compatibility, but it cannot by itself preserve dual trust across a process restart. The following rules complete the producer side of the Pack rotation contract without adding an endpoint:

  1. Every PackPublicKey ledger record persists its exact, non-secret publicMaterial, just as a SigningRoot record does. Before a legacy active Pack record that lacks this field is demoted, Platform captures the current materialized scalar into that record. If the scalar is absent, inconsistent, or not valid RSA public-key PEM, introduce fails before its first store write and requires operator recovery. Exactly one PUBLIC KEY or RSA PUBLIC KEY PEM block (plus surrounding whitespace) is accepted; private-key or mixed content is rejected before persistence so secret material cannot enter this non-secret ledger/response surface.
  2. A scalar-only legacy installation (no Pack ledger rows) remains readable. Platform publishes a synthetic active member named legacy-pem-sha256-<sha256-of-the-exact-UTF8-PEM-text>. On the first lifecycle introduce, that same identity and exact PEM are persisted as the retiring root before the new scalar replaces it; the new record’s supersedesKeyId points to the synthetic identity.
  3. The existing GET /platform/verification-settings/{service} response for PacksRegistry carries all three flat configuration values:
    • PacksRegistry:Verification:PublicKeyPem — unchanged active scalar;
    • PacksRegistry:Verification:TrustSetJson — canonical JSON {"schema":"stellaops.packsregistry.verification-trust-set/v1","regionProfile":"<profile>","roots":[{"keyId":"...","status":"active|retiring","publicKeyPem":"..."}]};
    • PacksRegistry:Verification:TrustEpoch — sha256: plus lowercase SHA-256 of UTF-8 without a BOM over <profile> + "\n" + <canonical-json>.
  4. Canonical roots use lowercase wire statuses and are sorted by ordinal keyId, then status. Every Pack keyId is stored in exact trimmed form, contains no control characters, and is at most 256 UTF-8 bytes. Introduce rejects an invalid id before any write, and publication fails closed when retained history violates that contract; operators must recover the exact history rather than truncate or relabel it. A real ledger must have exactly one active root; every retiring root is included. Missing or malformed ledger JSON, missing/invalid PEM, duplicate key ids or normalized RSA keys, case-insensitive key-id collisions or lifecycle-key variants, or an ordinal mismatch between the active ledger PEM and active scalar fails closed. No real-ledger member is silently omitted. Pack publication reads one direct coherent store snapshot on every pull, bypassing the general replica-local settings cache, so the first pull through any replica after invalidation commit cannot re-publish the revoked member.
  5. Live Pack verification includes active + at most one retiring root only. A distinct introduce is rejected atomically while a prior root remains retiring; the operator must finish re-enveloping and invalidate it before starting another rotation. An exact retry of the already committed active id + PEM remains idempotent. Pack artifacts are mutable and re-enveloped, so an archived Pack root is ledger/audit history, not live trust. Signing-root archived semantics remain unchanged: immutable/transparency-anchored material may continue to verify through archived signing-root JWKS members. Revoked roots are never published.
  6. Pack introduce is one conditional store mutation, not a sequence of independently visible setting writes. Platform captures an expected-state token over the scoped and legacy-unscoped active scalars, every Pack ledger row, and Pack re-envelope progress. PostgreSQL serializes by a stable profile-scoped transaction advisory lock, directly re-reads and compares that state after taking the lock, and commits retiring metadata, new active metadata, the scoped scalar, and the initial re-envelope status in one transaction. When a unique pre-existing lifecycle row uses noncanonical casing, updates and deletes target that exact physical key; only an absent row is created with canonical spelling. The in-memory implementation mirrors this under one gate. A changed token raises a typed conflict and writes nothing, so concurrent different-key introduces cannot create two active roots or choose the scalar by last writer. Retrying the exact active key id and ordinal-equal PEM after a lost response returns the committed result without another transition; the same key id with different material is rejected. A different key id is also rejected when its RSA key matches any retained Pack ledger member after canonical ExportSubjectPublicKeyInfo normalization and SHA-256, including alternate PKCS#1/SPKI PEM encodings and archived/revoked history. Lifecycle HTTP mutations surface the typed race as 409 with stable code verification_root_state_conflict; other validation remains 400.
  7. Introduce, invalidate, Pack re-envelope progress, and legacy bootstrap are the only application writers of those rows and all use the same conditional mutation. Generic environment-setting Set, batch Set, and Delete reject the unscoped/scoped Pack scalar, Pack metadata, and Pack re-envelope families. The existing admin PUT retains one narrow bootstrap: an exact Verification:<known-profile>:PacksRegistry:PublicKeyPem may establish a strictly validated scalar only while the Pack ledger is empty. Exact material retry is idempotent; replacement, unknown-profile/unscoped writes, metadata/progress writes, and every generic delete are rejected. Rejected generic lifecycle-owned PUT/DELETE calls return 409 verification_setting_lifecycle_owned; a strict-bootstrap conditional race returns 409 verification_root_state_conflict. Pre-existing unscoped legacy rows remain readable and are captured by lifecycle introduce.
  8. Direct Pack lifecycle snapshots are bounded before materialization: at most 256 retained root rows, at most 64 KiB UTF-8 per lifecycle value, and at most 4 MiB UTF-8 for the complete profile snapshot. PostgreSQL reads length columns first under sequential access and fetches at most one row beyond the row ceiling. Both supported mutation stores validate the complete projected state against the same bounds before their first write. Exceeding any bound fails publication and mutation closed for owner-approved recovery; no retained history is silently omitted or truncated.
  9. Invalidating the sole active Pack root deliberately leaves publication fail-closed; no retiring root is auto-promoted or rewritten. Recovery may introduce one cryptographically distinct new active root only when the real ledger has zero active members, no retiring member, and no scoped or unscoped active scalar row. Existing archived/revoked rows and their audit history remain unchanged and reserve their normalized RSA keys against relabeling/reintroduction. Any scalar-present or multi-active mismatch still requires operator recovery and writes nothing. A retained terminal row that predates publicMaterial retention also blocks introduce: its identity cannot be reconstructed safely, the generic settings API cannot repair it, and the operator must recover the exact original public PEM for an owner-approved audited ledger repair or keep rotation fail-closed.
  10. Platform and the startup consumer share one 64 KiB UTF-8 transport ceiling for the complete serialized PacksRegistry verification-settings response, not merely for each retained row. Introduce and legacy bootstrap serialize the exact projected outer dictionary with the endpoint JSON options before their first write; one byte over commits nothing. Publication independently applies the same check to retained state and fails closed for bounded operator recovery.

Offline-seed export trust addendum (2026-09-17)

The Pack trust publication addendum above is written around one material type and one algorithm. The offline-seed export key needs every rule in it and is neither. This addendum states how the mechanism generalizes and what stays different.

  1. A new material type PackExportPublicKey joins PackPublicKey, VexHmacTrustRoot and SigningRoot. It governs the ES256 (NIST P-256) key that signs catalog-manifest.json.dsse.json inside an offline-seed export archive, which the importing estate verifies before admitting the archive’s packs. Its blast radius is the reason it is a separate family rather than another root in the Pack set: a pack key admits one pack, this key admits an entire catalog, so no document may ever present both as interchangeable trust.
  2. Paragraphs 1–10 above apply unchanged to this family: retained publicMaterial, canonical ordering and bounded key ids, exactly one active plus at most one retiring live root, the expected-state token, the profile advisory lock, one-transaction commit, the 256-row / 64 KiB-value / 4 MiB-snapshot ceilings, no auto-promotion on invalidating the sole active root, and no publication of revoked roots. They are applied by one implementation parameterized by a trust-family descriptor (VerificationTrustFamily), not by a second copy. A copied implementation of fail-closed rules does not fail loudly when it drifts; it silently admits what the original refused.
  3. Exactly three things differ per family, and they are the descriptor’s contents:
    • ledger/progress namespace segment — pack-public-key vs pack-export-public-key;
    • materialized active scalar — Verification:<profile>:PacksRegistry:PublicKeyPem vs Verification:<profile>:PacksRegistry:Export:PublicKeyPem;
    • the algorithm the public material must parse as — RSA vs ES256. Published keys follow: PacksRegistry:Verification:{PublicKeyPem,TrustSetJson,TrustEpoch} and PacksRegistry:Export:Verification:{TrustSetJson,TrustEpoch}. The export names are exactly the section jobengine-web already binds, so the consumer needed no new configuration surface. The export family publishes no single-active compatibility scalar. Paragraph 3’s scalar exists for Pack consumers that predate the trust set; the export family has none, and the scalar would occupy the very configuration key an operator uses to configure a local key — leaving the consumer unable to distinguish a published root from a mounted one, which is exactly the ambiguity paragraph 8 below refuses. Pre-release, a new family gets no compatibility surface.
  4. The material validator is a sibling, not a reuse. Paragraph 1’s normalisation is RSA-specific (ExportSubjectPublicKeyInfo over an RSA key, PKCS#1 and SPKI labels both accepted). The export family accepts only the SPKI PUBLIC KEY label and requires the P-256 curve specifically: a P-384 key imports cleanly as an EC key and would then fail to verify every archive the estate holds, so the curve is part of the material rule and not merely of the signature suite. The non-secret guarantees are identical and are enforced for both families before any store write — one PEM block plus surrounding whitespace, no PRIVATE KEY content, value within the 64 KiB bound — and the fingerprint is still taken over ExportSubjectPublicKeyInfo, so alternate encodings of one key still collide and a revoked key still cannot be relabeled and reintroduced.
  5. Serialization boundary. The advisory lock stays profile-scoped, so the two families’ rotations serialize against each other; rotations are rare operator actions and one ordering is worth more than the concurrency. The expected-state token is family-scoped and hashes the family id, so a pack rotation does not spuriously conflict a concurrent export rotation, and — because the id is inside the hashed material — two empty families do not produce the same token, which would otherwise let a stale token from one satisfy the other’s compare-and-swap.
  6. Bounds. Paragraph 8’s row and byte ceilings now apply per family snapshot: each family is read, bounded and committed independently, so each carries the stated ceiling rather than sharing it. Paragraph 10’s 64 KiB transport ceiling is the one that protects the consumer, and it applies to the complete serialized verification-settings response — both families together — as well as to each family’s publication as it is built. Exceeding either fails publication closed.
  7. No legacy bootstrap for the export family. Paragraph 7’s narrow generic-PUT bootstrap exists to rescue scalar-only installations that predate the Pack ledger. The export family has no such installed base, so it has no bootstrap path and no synthetic legacy identity: its first root arrives through lifecycle introduce. Generic Set/Delete reject its rows exactly as they reject the Pack family’s, and that fencing is derived from the registered families rather than restated, so a future family is fenced off when it is registered rather than when someone remembers.
  8. Consumer rule (jobengine-web). PacksRegistry:Export:Verification:TrustSetJson already outranks a locally mounted PublicKeyPemFile, so a successful pull supersedes local configuration on its own. Configuring both at once is refused at startup. Resolving it by precedence would look correct while it ran and would let a key the operator believed they had revoked return the moment Platform sourcing was switched off; the failure being prevented is not “the wrong key is used today” but “the retired key comes back later”. An estate with no published export root keeps its local key and is unaffected, and an estate that never imports configures neither.

References