Runbook — HSM as the master KEK source

Audience: Platform / Integrations / ReleaseOrchestrator operators. Applies to: all three services that resolve the master KEK. Related runbooks: kek-rotation.md, kek-source-vault.md, credential-storage-algorithm-switch.md.

When to use this

Production Stella Ops deployments in regulated environments where the master KEK must be sealed behind a hardware security module (PKCS#11-compliant). The deployment already runs an HSM and has a wrapping key provisioned on it.

Hardware-validation caveat: the code path is production-ready but full functional validation requires actual HSM hardware. Operators using SoftHSM (CI-friendly substitute) get a working path; operators using Thales Luna / YubiHSM / equivalent get a code path that needs first-deployment validation against their hardware.

Service-wiring caveat (verified 2026-05-31): the HSM KEK source is library-complete and unit-tested, but no shipped service wires it yet — Platform, Integrations, and ReleaseOrchestrator each hardwire EnvKekSource and do not read Crypto:Kek:Source. Steps 3–4 below are the glue an operator (or a future sprint) must add before Crypto:Kek:Source=hsm takes effect. Treat this runbook as the wiring guide, not a description of out-of-the-box behaviour.

Prerequisites

Commissioning steps

1. Generate the master KEK

openssl rand -out /tmp/kek-bytes 32

2. Wrap the KEK with the HSM wrapping key

The exact command depends on the PKCS#11 toolchain. With pkcs11-tool + an AES-256 wrapping key labelled stellaops-kek-unwrap:

IV=$(openssl rand -hex 12)
echo $IV | xxd -r -p > /tmp/iv.bin

pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
  --label stellaops-kek-unwrap \
  --encrypt --mechanism AES-GCM \
  --iv-file /tmp/iv.bin \
  --input-file /tmp/kek-bytes \
  --output-file /tmp/kek-wrapped.bin

base64 < /tmp/kek-wrapped.bin   # -> WrappedKekBase64
base64 < /tmp/iv.bin            # -> IvBase64

Immediately delete the plaintext KEK: shred -u /tmp/kek-bytes. The HSM keeps the wrapping key; you only need the wrapped ciphertext + IV to boot.

3. Bridge StellaOps.Cryptography.Plugin.Hsm → IHsmKekUnwrapper

The IHsmKekUnwrapper interface is narrow on purpose so each operator can bridge to whatever HSM client they have.

Wiring status (verified against source 2026-05-31): the bridge below is a reference implementation you must add yourself — it is not checked into any service today. Program.cs in Platform, Integrations, and ReleaseOrchestrator all hardwire new EnvKekSource(...) (see src/Platform/StellaOps.Platform.WebService/Program.cs ~line 886 and the matching blocks in src/Integrations/StellaOps.Integrations.WebService/Program.cs and src/ReleaseOrchestrator/__Apps/StellaOps.ReleaseOrchestrator.WebApi/Program.cs). No shipped service calls ReadSelectedKekSourceKind, AddHsmKekSource, or registers an IHsmKekUnwrapper, and no HsmPluginKekUnwrapper type exists in the tree. The library half is complete and unit-tested (HsmKekSource, AddHsmKekSource, ReadSelectedKekSourceKind, IHsmKekUnwrapper); the service-side dispatch in step 4 is the remaining glue an operator (or a future sprint) supplies. Until that glue lands, setting Crypto:Kek:Source=hsm in config alone has no effect — the services ignore the key and continue resolving the KEK from the env fallback chain.

The reference bridge against the existing HSM plugin:

using StellaOps.Cryptography.CredentialStore.Hsm;
using StellaOps.Cryptography.Plugin.Hsm;

internal sealed class HsmPluginKekUnwrapper : IHsmKekUnwrapper
{
    private readonly IHsmClient _hsmClient;

    public HsmPluginKekUnwrapper(IHsmClient hsmClient)
    {
        _hsmClient = hsmClient;
    }

    public Task<byte[]> UnwrapAsync(
        string keyLabel,
        ReadOnlyMemory<byte> wrappedKek,
        string mechanism,
        ReadOnlyMemory<byte>? iv,
        CancellationToken cancellationToken)
    {
        var hsmMechanism = mechanism.ToUpperInvariant() switch
        {
            "AES-256-GCM" => HsmMechanism.Aes256Gcm,
            "AES-128-GCM" => HsmMechanism.Aes128Gcm,
            _ => throw new NotSupportedException(
                $"HSM unwrap mechanism '{mechanism}' is not supported by the bridge."),
        };

        return _hsmClient.DecryptAsync(
            keyLabel, wrappedKek.ToArray(), hsmMechanism, iv?.ToArray(), cancellationToken);
    }
}

Add this dispatch to each service’s Program.cs, replacing the current hardwired new EnvKekSource(...) registration (see the wiring-status note in step 3 — this block does not exist in the tree yet):

using StellaOps.Cryptography.CredentialStore;
using StellaOps.Cryptography.CredentialStore.Hsm;

var kind = KekSourceServiceCollectionExtensions.ReadSelectedKekSourceKind(builder.Configuration);
if (kind == "hsm")
{
    // The HSM plugin must have been loaded + started before this point.
    builder.Services.AddSingleton<IHsmKekUnwrapper, HsmPluginKekUnwrapper>();
    builder.Services.AddHsmKekSource(builder.Configuration);
}
// else: keep the existing EnvKekSource / FileKekSource / VaultKekSource branch.

ReadSelectedKekSourceKind returns env when Crypto:Kek:Source is unset (so omitting this dispatch preserves today’s env behaviour) and throws at registration time on any value outside env | file | vault | hsm.

4. Configure the service

The Crypto:Kek block below is verified against source: Crypto:Kek:Source is the literal key KekSourceServiceCollectionExtensions.ReadSelectedKekSourceKind reads (allowed values env | file | vault | hsm; unknown values throw at registration time), and Crypto:Kek:Hsm binds onto HsmKekSourceOptions (KeyLabel, WrappedKekBase64, Mechanism default AES-256-GCM, optional IvBase64).

{
  "Crypto": {
    "Kek": {
      "Source": "hsm",
      "Hsm": {
        "KeyLabel": "stellaops-kek-unwrap",
        "WrappedKekBase64": "<base64 from step 2>",
        "Mechanism": "AES-256-GCM",
        "IvBase64": "<base64 IV from step 2>"
      }
    }
  }
}

The HSM plugin itself (the thing your IHsmKekUnwrapper bridge talks to) is configured separately through the crypto plugin loader. The plugin binds an HsmOptions (LibraryPath, SlotId, TokenLabel, Pin, KeyIds, AllowSimulation, …; see src/Cryptography/StellaOps.Cryptography.Plugin.Hsm/ and plugin.yaml). The verified plugin-config root is StellaOps:Crypto:Plugins (AddStellaOpsCryptoFromConfiguration), with per-plugin keys driven by the loader convention — treat the section below as illustrative shape, not a verbatim key path, and confirm the exact section name against your deployment’s plugin loader before relying on it. KeyIds must include the wrapping key’s label so the plugin permits the unwrap operation.

// illustrative shape — confirm the plugin-loader section path for your deployment
{
  "LibraryPath": "/usr/lib/softhsm/libsofthsm2.so",
  "SlotId": 0,
  "TokenLabel": "stellaops",
  "Pin": "1234",
  "KeyIds": "stellaops-kek-unwrap"
}

5. Restart and verify

docker logs platform | grep "HSM KEK source resolved"
# Expect ONE line per service:
#   HSM KEK source resolved (kek_id=platform:connector-credentials:v1, key_label=stellaops-kek-unwrap, mechanism=AES-256-GCM).
# The plaintext KEK, the wrapped ciphertext, and the HSM PIN MUST NOT appear in any log line.

stella crypto kek status --kek-id platform:connector-credentials:v1
# Source should now report "hsm".

Hardware-validation matrix

HSMCode path validatedNotes
HsmKekSource ↔ IHsmKekUnwrapper contract (CI)YES (in-memory AES-GCM mock unwrapper)HsmKekSourceTests exercises happy-path unwrap, the in-process cache, no-silent-fallback on unwrap failure, and the sentinel-leak assertions — using a hand-rolled in-memory AesGcm unwrapper, not a real HSM.
SoftHSM 2NO (not run in CI)Documented here as the CI-friendly substitute, but HsmKekSourceTests does not drive a SoftHSM binary (CI hosts do not reliably ship one). Operators wiring SoftHSM get a working code path that still needs a one-time deployment check (see below).
YubiHSM 2NO (no hardware in CI)Code path expected to work via the existing Pkcs11HsmClient (real CKM_AES_GCM decrypt through Pkcs11HsmClientImpl); first-deployment validation required.
Thales LunaNO (no hardware in CI)Code path expected to work via the existing Pkcs11HsmClient; first-deployment validation required.
AWS CloudHSMNO (cloud KMS deliberately not shipped)Operators install the AWS PKCS#11 library + treat as a generic PKCS#11 token; IHsmKekUnwrapper bridge unchanged.

For first deployment against new HSM hardware, the recommended validation is:

  1. Wrap a throwaway 32-byte secret via step 2.
  2. Run the service with Crypto:Kek:Source=hsm pointed at it.
  3. Verify the startup log shows HSM KEK source resolved with the correct key label.
  4. Verify stella crypto kek status reports source=hsm.
  5. Round-trip a single connector credential write + read to verify the AEAD layer is decrypting against the HSM-unwrapped KEK.

Failure modes

Symptom (startup log)CauseOperator action
HSM KEK source: Crypto:Kek:Hsm:KeyLabel is required.KeyLabel unset (constructor validation)Set Crypto:Kek:Hsm:KeyLabel.
HSM KEK source: Crypto:Kek:Hsm:WrappedKekBase64 is required.WrappedKekBase64 unset (constructor validation)Set Crypto:Kek:Hsm:WrappedKekBase64 to the step 2 output.
HSM KEK source: Crypto:Kek:Hsm:Mechanism is required.Mechanism blanked out (constructor validation; the option defaults to AES-256-GCM, so this only fires if explicitly emptied)Restore a mechanism value.
HSM KEK source: Crypto:Kek:Hsm:WrappedKekBase64 is not valid base64.Wrapped-KEK config typoRe-run step 2; verify base64 -d round-trips.
HSM KEK source: Crypto:Kek:Hsm:IvBase64 is not valid base64.IV config typoRe-run step 2; verify the IV base64 -d round-trips.
HSM KEK source: unwrap failed via key 'X' with mechanism 'Y'. CKR_KEY_HANDLE_INVALIDWrong key label or wrapping key absentVerify key exists on the HSM token; check pkcs11-tool --list-objects.
HSM KEK source: unwrap failed via key 'X' with mechanism 'Y'. CKR_PIN_INCORRECTHSM PIN wrongFix the HSM plugin’s Pin (see step 4). Service WILL NOT start until resolved.
HSM KEK source: unwrap failed via key 'X' with mechanism 'Y'. CKR_MECHANISM_INVALIDHSM does not implement the configured mechanismSwitch to a mechanism the HSM supports (re-wrap KEK + update config).

The unwrap failed via key '...' with mechanism '...' prefix is emitted verbatim by HsmKekSource; the trailing text (CKR_*) is the unwrapper’s own message and propagates through unchanged. The wrapped-KEK ciphertext is never included in the message.

The source fails-startup on any unwrap failure — no silent fallback to env-var.

Audit & sentinel guarantees