ADR-007: KEK Lifecycle and Rotation

Status: Accepted Date: 2026-05-27 Sprint: SPRINT_20260529_028_Key_management_operations.md (RP-028-001…008, RP-028-014) Related ADRs: ADR-004 (forward-only migrations), ADR-005 (single-operator multi-environment tenancy), ADR-006 (execution plugin framework — same plugin-host substrate the HSM source bridges through) Related runbooks: KEK rotation, KEK source: Vault, KEK source: HSM, credential-storage algorithm switch

This record makes the deployment-secret master KEK rotatable, observable, and source-pluggable (env, file, Vault, HSM) without downtime or data loss, and audits every rotation. Read it before touching any credential-ciphertext store, the KEK source configuration, or the crypto-control surface. Operators rotating a live KEK should follow the KEK rotation runbook.

Implementation status (2026-07-18). The accepted architecture below is still the target, not the complete current runtime contract. The wired Platform launcher passes the same AEAD instance as both the old and new key, then records a new version with the active source and fingerprint. Current operations therefore perform a resumable current-material re-seal/version bump; they do not apply a selected source, source reference, grace window, or changed KEK material. Platform responses expose this truth through rotationSemantics (operationMode=current_material_reseal, changesKeyMaterial=false, and all source/reference/grace applied flags false). The CLI source/reference/grace options and the console wizard inputs must not be treated as applied until a coordinated Platform + Integrations + ReleaseOrchestrator successor supplies both old and new key resolvers, a dual-key grace lifecycle, and forcing-function proof of changed fingerprints, old/new ciphertext openability, and retirement after grace. See the rotation runbook for the operational limit.

Context

Sprint 027 shipped the dual-mode deployment-secret crypto stack. Three ciphertext stores share a single master KEK (“key-encryption key”) protecting their per-row data-encryption keys:

  1. Platform — platform.connector_credentials (operator-entered SCM / registry / Vault / chat credentials)
  2. Integrations — integrations.integrations inline credentials (the in-process equivalent for connector plugins)
  3. ReleaseOrchestrator — release_orchestrator.deployment_bundles (push-mode bundles containing per-deployment secrets)

After Sprint 027 the master KEK was resolved from a single env var (STELLAOPS_BOOTSTRAP_KEY and per-store fallbacks) at process start. This left four gaps:

Sprint 027’s Decisions & Risks called this out explicitly as the lone unfinished thread blocking production-grade adoption of the dual-mode crypto stack in regulated environments.

Decision

Introduce a pluggable IKekSourceabstraction that materialises the master KEK at process start and provides a re-resolve hook for rotation. Track KEK versions in a new crypto.kek_versionsledger table — one row per (kek_id, version) — that records the source, state, non-secret fingerprint, activator, and timestamp for every version that has ever been active. Drive multi-store re-seal through a shared CredentialReSealWalkerlibrary that re-seals every row across the three stores idempotently and resumably. Expose the operator-control surface through Platform admin endpoints, the stella crypto kek CLI subgroup, and the console-admin /console-admin/crypto-control UI. Every rotation event is audited. KEK material never enters audit, logs, UI payloads, HTTP responses, or any exception that escapes the seal/open call boundary.

IKekSource shapes

record KekMaterial(string KekId, int KekVersion,
                   ReadOnlyMemory<byte> SecretBytes,
                   DateTimeOffset? ExpiresAt);

interface IKekSource
{
    Task<KekMaterial> ResolveAsync(KekSourceContext ctx, CancellationToken ct);
}

Production-ready sources in this sprint:

SourceLibraryPurpose
envStellaOps.Cryptography.CredentialStore (EnvKekSource)Default; byte-identical to Sprint 027 resolution.
fileStellaOps.Cryptography.CredentialStore (FileKekSource)Cheap hardening; owner-only ACL enforced.
vaultStellaOps.Cryptography.CredentialStore.Vault (flagship)Reuses operator’s existing on-prem HashiCorp Vault.
hsmStellaOps.Cryptography.CredentialStore.HsmPKCS#11 unwrap via StellaOps.Cryptography.Plugin.Hsm.

Source selection is driven by the Crypto:Kek:Source config key. Unknown values fail at registration (no silent downgrade). Default = env, preserving Sprint 027 behaviour.

crypto.kek_versions ledger

CREATE TABLE IF NOT EXISTS crypto.kek_versions (
    kek_id            VARCHAR(80)  NOT NULL,
    version           INT          NOT NULL,
    state             VARCHAR(20)  NOT NULL,    -- active | grace | retired | rotating
    source            VARCHAR(40)  NOT NULL,    -- env | file | vault | hsm
    source_metadata   JSONB        NULL,        -- e.g. {"vault_path":"secret/stellaops/kek/v2"}
    activated_at      TIMESTAMPTZ  NOT NULL,
    retired_at        TIMESTAMPTZ  NULL,
    activated_by      VARCHAR(160) NOT NULL,
    fingerprint       VARCHAR(64)  NOT NULL,    -- non-secret; 16-hex truncation of double-SHA-256
    PRIMARY KEY (kek_id, version)
);

The fingerprint is a one-way 64-bit truncation of a double-SHA-256 with a KDF label binding (KekFingerprint.Compute). It is non-secret and used to let operators eyeball “is this the KEK I think it is?” without ever rendering KEK material. The KEK bytes themselves never enter the ledger.

The shared library StellaOps.Cryptography.CredentialStore.Persistence ships the schema as an embedded resource so all three services migrate identically. Migration is forward-only per ADR-004.

Per-row sealed-by-KEK-version + algorithm tracking

Every ciphertext row carries (kek_id, kek_version, algorithm). Sprint 027 rows defaulted to (kek_id=<resolver-derived>, kek_version=1, algorithm=AES-256-GCM); the additive migrations (Platform 080, Integrations 004, ReleaseOrchestrator 013) backfill the defaults on existing rows. This gives the walker a deterministic predicate (secret_kek_version < <active>) for “rows still pinned to an older KEK” and makes per-version blob population queryable without scanning ciphertext.

Multi-store re-seal walker

CredentialReSealWalker operates over the abstract ICredentialReSealStore port (CountByKekVersionAsync + ReSealBatchAsync). Each of the three services implements the port over its own ciphertext store:

All three use SELECT ... FOR UPDATE SKIP LOCKED for safe concurrency, scrub per-row plaintext in finally, and perform an atomic swap of (nonce, ciphertext, kek_version, alg, kek_id). Progress is persisted in crypto.reseal_progress so a crashed walker resumes automatically via CredentialReSealResumeHostedService. Batch=256 (configurable), per-batch timeout 30s, rate-limited (default 100 batches/sec) so an O(million)-row rotation can be wall-clock-windowed.

Control plane

Consequences

Positive

Negative

Neutral

Alternatives considered (rejected)

Keep env-var-only; operators rotate via stop / swap-env / restart + custom one-time scripts

Rejected. Not auditable, not safe at scale, and blocks regulated-environment adoption. The “I edited the env var on three hosts and restarted” workflow provides no record of who, when, or which version, and offers no protection against forgetting one of the three ciphertext stores. Every Sprint 027 operator who asked about rotation hit this wall.

Rotation via one-shot Kubernetes Job

Rejected. Ties the control plane to a deployment model the operator may not use — Stella Ops targets non-Kubernetes estates. A CLI-driven control plane is portable across docker-compose, Nomad, bare-metal systemd, and the operator’s hosted Kubernetes (which can call the CLI from a Job if they so choose). The reverse — making the control plane K8s-native — is not portable.

AWS KMS / Azure Key Vault as production sources or DI-registered stubs

Rejected. Stella Ops targets self-hosted, offline/air-gap-capable on-premises estates. Cloud-managed KMS contradicts that posture and was the explicit operator decision recorded in the Sprint 028 dossier (2026-05-27). The IKekSource interface remains open for downstream installs that need it — they implement the plugin in their own assembly. Shipping a stub that throws NotImplementedException would imply support not delivered and appear in install scans as a “supported but unconfigured” source, which is the inverse of the truth.

Auto-scheduled rotation

Deferred. Rotation is a high-trust operation; the operator should drive cadence in this generation of the surface. The schema, walker, and audit trail are all rotation-id-driven, so scheduling can be layered on top later without re-doing the foundation.

KEK material in crypto.kek_versions (encrypted under an HSM-only KEK)

Rejected. Even encrypted-at-rest, this would put KEK material on the admin DB’s backup tape, in PITR archives, and in operator psql sessions. The ledger holds only metadata + a non-secret fingerprint; the KEK material lives exclusively in the configured IKekSource and in process memory.

Per-tenant KEK isolation

Out of scope. ADR-005 establishes that per-tenant isolation is achieved cryptographically via the per-tenant HKDF DEK derivation. The master KEK is per-installation. Per-tenant KEK is desirable but is its own sprint (key custody, ceremony tooling, tenant-lifecycle integration).

References