Runbook — Master KEK rotation

Audience: Platform / Integrations / ReleaseOrchestrator operators. Applies to: all three services that resolve the master KEK (platform.connector_credentials, integrations.integrations, release_orchestrator.deployment_bundles). Related ADRs: ADR-007 (KEK lifecycle and rotation), ADR-008 (regional algorithm routing). Related runbooks: kek-source-vault.md, kek-source-hsm.md, credential-storage-algorithm-switch.md.

Implementation status (verify against src/). The KEK control plane (status / history / dry-run plan / rotate / abort + progress streaming) is live: CLI stella crypto kek (src/Cli/StellaOps.Cli/Commands/CryptoCommandGroup.cs), Platform admin endpoints (src/Platform/StellaOps.Platform.WebService/Endpoints/CryptoKekAdminEndpoints.cs), walker (src/__Libraries/StellaOps.Cryptography.CredentialStore.ReSeal/). What is NOT yet wired (Phase 2, RP-028-008): per-rotation source switch and grace-hours configuration. The CLI accepts --new-source, --new-source-ref, and --grace-hours so operator scripts can be staged, but the handler discards them today (HandleKekRotateAsync does _ = newSourceRef; / _ = graceHours;). In the current wired path the walker re-seals rows under byte-identical KEK material (oldAead and newAead are the same instance — see BackgroundLocalRotationWalkerLauncher), so a rotation is a kek_versions version bump, NOT a material/source change. Sections below flag the forward-looking behaviour explicitly. The roadmap target is in ADR-007.

Required scopes. Read verbs (status / history / rotation-plan / progress) require crypto:kek:read; mutating verbs (rotate / abort) require crypto:kek:rotate (catalog: src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs → PlatformCryptoKekRead / PlatformCryptoKekRotate). These are distinct from crypto:admin so rotation authority can be granted independently. Policy composition (verify against src/): the read policy (PlatformPolicies.CryptoKekRead) is satisfied by any of crypto:kek:read, crypto:kek:rotate (rotation implies read, so a rotation operator does not need a separate read grant), or ops.admin; the rotate policy (PlatformPolicies.CryptoKekRotate) is satisfied by crypto:kek:rotate or ops.admin. ops.admin is the platform-operator escape hatch for both (AddStellaOpsAnyScopePolicy registrations, src/Platform/StellaOps.Platform.WebService/Program.cs lines 242–243; policy/scope constants in src/Platform/StellaOps.Platform.WebService/Constants/PlatformPolicies.cs / PlatformScopes.cs).

When to rotate

TriggerCadence
Scheduled hygieneAnnual at minimum; quarterly for high-assurance posture.
Post-incidentAfter any suspected exposure of the master KEK or any custodian credential (Vault token, AppRole secret_id, HSM PIN, file-source ACL breach).
Post-personnel-changeWhen a custodian with access to the source material leaves the team.
Post-source-changeWhen migrating between sources (env → file → vault → hsm). Source-switch-as-rotation is Phase 2 (RP-028-008); today the --new-source flag is accepted but discarded — see the implementation status note. Until then, treat a source migration as an out-of-band re-key plus a fresh rotation, and do not skip the dry-run.
Compliance audit findingWhenever the auditor requires a fresh KEK against the production data set.

Before you start

Pre-rotation checks

  1. Confirm current state.

    stella crypto kek status --kek-id platform:connector-credentials:v1

    You should see exactly one active version. The concurrent-rotation guard keys off the per-store crypto.reseal_progress ledger, not the version ledger: the endpoint refuses a new rotation while any store has a running row for this kek_id (ListActiveByKekAsync), returning HTTP 409. If a prior rotation is still in flight (or stuck running after a crash), finish or abort it first.

  2. Confirm source reachability.

    • vault: vault token lookup against the configured auth principal; vault kv get <mount>/<path> to confirm the next KEK material is readable.
    • hsm: pkcs11-tool --list-objects against the configured slot to confirm the wrapping key is still present and the wrapped KEK ciphertext is the one you intend to roll to.
    • file: ls -ln <path> to confirm owner-only ACL (POSIX 0600 / Windows owner-only DACL).
    • env: confirm the new env var is set in the deployment manifest for every service that resolves the KEK.

  3. Confirm stella crypto profile validate returns ok. If validation surfaces no_plugin_matches or fips_disagreement, fix the underlying issue before rotating — the post-rotation walker dispatches via the same primitive, and a broken primitive means the rotation will fail mid-way.

  4. Estimate the blob population.

    stella crypto kek status --kek-id <id> --format json | jq '.populations'

    .populations.stores.<store>.<version> is the per-(store, version) row count. At default Crypto:Reseal:BatchSize=256 and rate-limit Crypto:Reseal:MaxBatchesPerSecond=100 (CredentialReSealOptions, bound from Crypto:Reseal:*), the walker caps at ~25,600 rows/sec/store. There is also a per-batch Crypto:Reseal:BatchTimeout (default 30s); a batch exceeding it transitions the rotation to failed. (--grace-hours is accepted but not yet honoured — see the implementation status note above.)

Dry-run first

The dry-run reads counts and emits a RotateKekDryRun audit event but mutates nothing. The CLI flags --dry-run and --confirm are mutually exclusive (passing both, or neither, is rejected with exit code 2). --kek-id is required.

stella crypto kek rotate \
  --kek-id platform:connector-credentials:v1 \
  --dry-run

The CLI hits GET /api/v1/admin/crypto/kek/{kekId}/rotation-plan and prints:

KEK rotation plan: platform:connector-credentials:v1
  Current version:  3
  Proposed version: 4
  Batch size:       256
  Rows to convert:  1670

Per-store populations:
  platform.connector_credentials:
    v3: 1247 rows
  integrations.integrations:
    v3: 381 rows
  release_orchestrator.bundles:
    v3: 42 rows

(dry-run: nothing mutated. Re-run with --confirm to begin rotation.)

Interpret the plan:

Not emitted by the current dry-run plan: fingerprint, new_source_metadata, grace_hours, or an estimated wall-clock. The endpoint (CryptoKekAdminEndpoints.MapGet(".../rotation-plan")) returns only CurrentVersion, ProposedVersion, BatchSize, RowsToConvert, and populations. Because the wired rotation reuses the active version’s source and fingerprint (no material switch yet), there is no new fingerprint to eyeball. When the Phase 2 source switch (RP-028-008) lands, the plan is expected to carry the resolved new fingerprint for the STOP-on-mismatch check described in ADR-007.

Commit the rotation

stella crypto kek rotate \
  --kek-id platform:connector-credentials:v1 \
  --confirm

The CLI POSTs .../{kekId}/rotate (returns 202 Accepted with the new rotation_id), then polls .../rotation/{rotationId}/progress once per second and streams the aggregate per-store state to stdout. Expected output:

Rotation started: rotation_id=4f7e8a1c-...
  Old version: 3 → New version: 4
  Local store: platform.connector_credentials
  Downstream integrations: started (store=integrations.integrations)
  Downstream release-orchestrator: started (store=release_orchestrator.bundles)

Streaming rotation progress (Ctrl+C to detach; abort with `stella crypto kek abort`):
  platform.connector_credentials            running    768/1247 (61%)
  integrations.integrations                 completed  381/381 (100%)
  release_orchestrator.bundles              completed  42/42 (100%)
  ...
  platform.connector_credentials            completed  1247/1247 (100%)

Rotation complete.

The --new-source / --new-source-ref / --grace-hours flags shown in earlier drafts are accepted but discarded today (Phase 2 — see the implementation status note). The wired --confirm path bumps the kek_versions version with byte-identical material; it does not read a new source. A concurrent rotation against the same kek_id is rejected by the endpoint with HTTP 409; the CLI surfaces that as a non-zero exit.

Exit codes (streaming --confirm path):

On aborted or failed, the crypto.kek_versions ledger is left unchanged (the old version stays active); the failure detail lives on the per-store crypto.reseal_progress row (state='failed', sanitised last_error). No new kek_versions row is registered until the rotation completes (CryptoKekControlPlaneService.OnWalkerTerminalAsync).

Monitor progress

While the CLI streams, the same progress is available via:

# Server-side aggregate (Platform's view of all three stores).
# $TOKEN must carry the crypto:kek:read scope.
curl -sH "Authorization: Bearer $TOKEN" \
  https://stella-ops.local/api/v1/admin/crypto/kek/<id>/rotation/<rotation_id>/progress

# Console-admin UI
# Navigate to /console-admin/crypto-control/<kekId> (route guard requires
# crypto:kek:read). The detail page shows the reverse-chrono Rotation history;
# the dedicated rotate wizard at /console-admin/crypto-control/<kekId>/rotate
# (guarded by crypto:kek:rotate) polls progress live (~2s tick) and renders the
# per-store progress until a terminal state.

The UI polls the same progress endpoint on a fixed tick and renders the per-store completed/total. Watch each store advance to completed; a store reporting failed is the terminal signal to investigate last_error.

Abort

If a rotation goes sideways (wrong source picked, wrong fingerprint, source reachability collapsed mid-walk), abort it:

stella crypto kek abort \
  --kek-id platform:connector-credentials:v1 \
  --rotation-id 4f7e8a1c-...

Both --kek-id and --rotation-id are required; the CLI validates that --rotation-id is a well-formed GUID before calling the endpoint (POST .../rotation/{rotationId}/abort, scope crypto:kek:rotate).

Abort semantics:

After abort, the per-version blob populations show a mix: rows the walker already advanced sit on the new version number; the rest stay on the old version. In the current wired path this is benign because both versions share byte-identical KEK material, so every row opens regardless. (When the Phase 2 material switch lands, a partial rotation will require BOTH keys to be loaded to open both populations — see ADR-007 and the grace-window discussion below.) Plan a follow-up rotation to converge all rows onto one version. Audit event: RotateKekAbort.

Recovery from mid-rotation crash

The walker is resumable by design: crypto.reseal_progress checkpoints completed_rows / last_row_id atomically after each successful batch, and the re-seal query selects WHERE kek_version = oldVersion, which naturally excludes already-converted rows. Re-instantiating a walker with the same rotation_id picks up where the crash left off without re-processing converted rows (CredentialReSealWalker / 002_reseal_progress.sql).

Verify against src/. There is currently no automatic resume-on-startup hosted service (no CredentialReSealResumeHostedService in the tree), and no --resume flag on stella crypto kek rotate. The CLI rotate command accepts only --dry-run / --confirm (plus the discarded Phase 2 source/grace flags). Until an auto-resume launcher is wired, recovery after a crash is: confirm the in-flight row via stella crypto kek status --kek-id <id> (a stuck running reseal_progress row indicates an interrupted walker), then either re-trigger a rotation against the same kek_id (the concurrent-rotation guard refuses a second start while a running row exists, so abort the stale row first if it will never resume) or abort and start fresh. Track auto-resume in the RP-028 sprint dossier before relying on it.

In the future Phase 2 path, the resume would re-resolve OLD and NEW KEK material from the crypto.kek_versions source / source_metadata; if the OLD KEK source has been deleted (env var removed, Vault path gone) the resume cannot read pre-rotation rows. Treat the OLD source as immutable until the rotation has completed and been verified.

Post-rotation verification

# 1. Status should show v4 active and v3 retired.
#    (On successful completion the walker registers v4 as `active` and
#    transitions v3 directly to `retired` — there is no automatic intermediate
#    `grace` transition in the wired path; the `grace` state exists in the
#    enum but the completion handler retires the old version straight away.)
stella crypto kek status --kek-id <id>

# 2. Blob populations should show 100% on v4 across all three stores.
stella crypto kek status --kek-id <id> --format json | jq '.populations'

# 3. Audit log should show: DryRun (from the dry-run) → Begin → Abort (only if aborted).
#    Filter by rotation_id in your audit sink. NOTE: the per-step and completion
#    audit constants (rotate_kek_step / rotate_kek_complete) are declared in
#    AuditActions.cs but are NOT emitted by any code today — only
#    rotate_kek_dry_run / rotate_kek_begin / rotate_kek_abort are emitted
#    (CryptoKekAdminEndpoints).

# 4. A sanity-open of a random row via the Platform API confirms decryption.
# (Not a routine step — only if a deeper smoke is required.)

After the grace window (Phase 2 — roadmap)

Not yet implemented — verify against src/ before relying on this. The grace-window machinery described here is the ADR-007 target, not current behaviour. Today there is no grace_hours honouring, no time-based Grace→Retired transition, and no per-rotation in-memory retention window: on successful completion the walker registers the new version active and transitions the old version straight to retired (CryptoKekControlPlaneService.OnWalkerTerminalAsync). Because the wired path reuses byte-identical KEK material, there is no second key to purge from process memory and a restart has no KEK-lifecycle effect.

When the Phase 2 material switch (RP-028-008) lands, the intended behaviour is: once grace_hours has elapsed past the rotation completion, the OLD KEK is no longer needed in process memory; the OLD crypto.kek_versions row transitions grace → retired; and a scheduled service restart on each consuming service purges the OLD KEK material from process state. Until then, treat this section as forward-looking.

FAQ

“I rotated and now some rows can’t be opened — what do I do?”

In the current wired path this cannot happen from rotation alone: rotation reuses byte-identical KEK material, so every row opens regardless of which version it carries. A row that genuinely cannot be opened means the underlying KEK material itself was lost (env var / Vault path / HSM object deleted), not a rotation bug.

When the Phase 2 material switch (RP-028-008) lands, an unreadable row after a real rekey would point to one of: the OLD KEK source material deleted before re-seal finished; a service restarted mid-rotation dropping the OLD key; or a grace window shorter than the walker run. The recovery posture in that scenario is restore from backup — the ciphertext is unreadable without the original KEK and there is no in-system recovery.

Prevention (applies once a real material switch is wired):

“Can I rotate to a different source (env → vault)?”

Not yet. The CLI accepts --new-source / --new-source-ref, but the handler discards them (_ = newSourceRef;) — source switching is Phase 2 (RP-028-008). Today every rotation re-seals under the current source’s byte-identical material (a version bump, not a material change). When the source switch lands the intent is “read under the OLD source, write under the NEW source for the duration of the rotation”; track RP-028-008 before scripting against it.

“Can two kek_ids rotate simultaneously?”

Yes. Concurrency is per-kek_id; different kek_ids rotate independently.

“How do I rotate the KEK on an air-gap install?”

Same procedure. The CLI talks only to the Platform admin endpoint over the internal network; no external connectivity required. The new KEK source (Vault / HSM / file) must be reachable from the deployment’s internal network.

“How do I re-seal existing rows when changing the compliance profile / algorithm?”

Use the single-invocation workflow on stella crypto profile set --reseal. It persists the tenant profile and then triggers a KEK rotation whose targetAlgorithm is derived from the profile (MapProfileToAlgorithm: sm → SM4-GCM, gost → GOST-GCM, everything else → AES-256-GCM), forwarded to the rotate endpoint as ?targetAlgorithm=. The endpoint validates the algorithm against the configured ICredentialAeadPrimitive.SupportsAlgorithm and returns HTTP 400 (unsupported_target_algorithm) without mutating state if it is not supported. --reseal-kek-id overrides the default platform:connector-credentials:v1. This is the implemented forcing-function for migrating existing tenant rows to a new algorithm; see credential-storage-algorithm-switch.md.

Audit & sentinel guarantees