Runbook — Vault as the master KEK source

Audience: Platform / Integrations / ReleaseOrchestrator operators. Applies to: all three services that resolve the master KEK (platform.connector_credentials, integrations.integrations, release_orchestrator.deployment_bundles). Related runbooks: kek-rotation.md, kek-source-hsm.md, credential-storage-algorithm-switch.md. Status: Library complete (VaultKekSource, AddVaultKekSource, ReadSelectedKekSourceKind are implemented and unit-tested). Per-service selection of Crypto:Kek:Source=vault is not yet wired — Platform, Integrations, and ReleaseOrchestrator currently register EnvKekSource unconditionally (RP-028-008 Phase 2 lands the dispatch). This runbook is the commissioning procedure for the day that switch is flipped; the config it produces is inert until then. Forward-looking sections are flagged inline.

When to use this

Production Stella Ops deployments where the master KEK must be gated behind HashiCorp Vault’s access control + audit trail (instead of a plain env var). Operators already running Vault for Sprint 027 pull-mode AuthRef get this for free — the same Vault server can host the KEK at a distinct path.

Not for: cloud-KMS-only installs (AWS / Azure). Stella Ops targets self-hosted / on-prem and ships no KMS plugin. Operators landing in those clouds implement IKekSource directly — the interface is open.

Prerequisites

Commissioning steps

1. Generate the KEK

# 32 bytes of randomness, UTF-8 string for byte-identity with Sprint 027 env-var resolution.
openssl rand -base64 32 > /tmp/kek.b64

2. Store the KEK in Vault

Enable KV v2 if not already (most operators use the default secret mount):

vault secrets enable -path=secret kv-v2 2>/dev/null || true

Write the KEK as a single field. Choose the encoding to match Crypto:Kek:Vault:Encoding:

# UTF-8 (default — preserves Sprint 027 byte-identity wrt env-var migration)
vault kv put secret/stellaops/kek kek="$(cat /tmp/kek.b64)"

# base64 (operator preference; configure Crypto:Kek:Vault:Encoding=base64)
vault kv put secret/stellaops/kek kek="$(base64 < /tmp/raw-kek-bytes)"

# hex (operator preference; configure Crypto:Kek:Vault:Encoding=hex)
vault kv put secret/stellaops/kek kek="$(xxd -p -c 256 < /tmp/raw-kek-bytes | tr -d '\n')"

3. Create an auth principal with read capability

# Policy: read-only on the KEK path.
cat <<EOF | vault policy write stellaops-kek-reader -
path "secret/data/stellaops/kek" {
  capabilities = ["read"]
}
EOF

# AppRole binding (recommended for non-Kubernetes deployments)
vault auth enable approle 2>/dev/null || true
vault write auth/approle/role/stellaops-kek policies="stellaops-kek-reader"
ROLE_ID=$(vault read -field=role_id auth/approle/role/stellaops-kek/role-id)
SECRET_ID=$(vault write -field=secret_id -force auth/approle/role/stellaops-kek/secret-id)

Record ROLE_ID and SECRET_ID for the service config below.

4. Configure the service

In each consuming service’s appsettings.json or env:

{
  "Crypto": {
    "Kek": {
      "Source": "vault",
      "Vault": {
        "Address": "https://vault.internal:8200",
        "Path": "stellaops/kek",
        "Mount": "secret",
        "Field": "kek",
        "Encoding": "utf8",
        "StartupRetryWindow": "00:01:00",
        "Auth": {
          "Mode": "approle",
          "RoleId": "<ROLE_ID from step 3>",
          "SecretId": "<SECRET_ID from step 3>"
        }
      }
    }
  }
}

NOT YET WIRED IN THE SHIPPING SERVICES (as of SPRINT_20260529_028 Phase 1). The IKekSource abstraction, VaultKekSource, the AddVaultKekSource extension, and the ReadSelectedKekSourceKind source selector are all implemented and unit-tested in the libraries (StellaOps.Cryptography.CredentialStore / StellaOps.Cryptography.CredentialStore.Vault). However, the three consuming services do NOT yet dispatch on Crypto:Kek:Source — Platform (src/Platform/StellaOps.Platform.WebService/Program.cs), Integrations, and ReleaseOrchestrator all register EnvKekSource directly and unconditionally at startup. Setting Crypto:Kek:Source=vault in config therefore has no effect in the current builds; the env-var KEK is always used. The per-service dispatch below is the target wiring that RP-028-008 Phase 2 lands; until then this runbook commissions Vault for the day the switch is flipped, not for a live vault source.

The target per-service Program.cs wiring (replaces the unconditional EnvKekSource registration once Phase 2 lands):

using StellaOps.Cryptography.CredentialStore;
using StellaOps.Cryptography.CredentialStore.Vault;

var kind = KekSourceServiceCollectionExtensions.ReadSelectedKekSourceKind(builder.Configuration);
switch (kind)
{
    case "env":
        // NB: AddEnvKekSource requires the service's fallback chain
        // (IReadOnlyList<string> of config keys) as a second argument.
        builder.Services.AddEnvKekSource(builder.Configuration, /* fallbackChain */);
        break;
    case "file":
        builder.Services.AddFileKekSource(builder.Configuration);
        break;
    case "vault":
        builder.Services.AddVaultKekSource(builder.Configuration);
        break;
    case "hsm":
        // See docs/runbooks/kek-source-hsm.md. AddHsmKekSource takes only
        // IConfiguration; the service must separately register an
        // IHsmKekUnwrapper (bridge to the PKCS#11 plugin) beforehand.
        builder.Services.AddHsmKekSource(builder.Configuration);
        break;
}

ReadSelectedKekSourceKind throws at startup on any value other than env / file / vault / hsm (fail-fast over a quiet downgrade); unset defaults to env.

5. Restart and verify

# Tail the service logs during startup. You should see ONE line per service:
#   Vault KEK source resolved (kek_id=platform:connector-credentials:v1, path=secret/stellaops/kek, field=kek).
# The KEK BYTES, the Vault token, and the wrapped material MUST NOT appear in any log line.

docker logs platform | grep "Vault KEK source resolved"

# Verify via the admin endpoint (RP-028-005):
stella crypto kek status --kek-id platform:connector-credentials:v1
# Source should now report "vault"; fingerprint should match the new KEK.

The kek status verb maps to GET /api/v1/admin/crypto/kek/{kekId}/status on the Platform service (see PlatformCryptoKekClient). The CLI principal (token / client-credentials) must carry an authorising scope or the call returns 403:

Note: the ops-admin scope value is ops.admin (dot form, PlatformScopes.OpsAdmin), NOT ops:admin. It is a Platform-local scope constant and is not present in the canonical StellaOpsScopes.cs catalog (unlike the two KEK scopes below).

The two KEK scopes are defined in the canonical catalog (StellaOpsScopes.PlatformCryptoKekRead = crypto:kek:read, StellaOpsScopes.PlatformCryptoKekRotate = crypto:kek:rotate, StellaOpsScopes.cs) and are deliberately distinct from crypto:admin so KEK rotation can be granted without provider-catalogue mutation.

Caveat: because no shipping service dispatches on Crypto:Kek:Source yet (see the note in step 4), kek status reports the source recorded at KEK bootstrap, which today is always env regardless of the Crypto:Kek:Vault config above. It will not report vault until the Phase 2 per-service wiring lands.

Auth modes

token (development only)

{ "Mode": "token", "Token": "hvs.CAESI...-dev-token" }

Use for local development. Production should use AppRole or Kubernetes.

approle (recommended for non-Kubernetes)

{ "Mode": "approle", "RoleId": "...", "SecretId": "..." }

Vault issues a short-lived client token via the AppRole login endpoint; the source re-authenticates automatically on 403 (token revoked / expired). Lease duration is honoured (80% threshold for re-auth).

kubernetes (recommended for K8s deployments)

{
  "Mode": "kubernetes",
  "KubernetesRole": "stellaops-kek",
  "KubernetesTokenPath": "/var/run/secrets/kubernetes.io/serviceaccount/token"
}

The pod’s service account JWT is exchanged for a Vault token via the Vault Kubernetes auth method. The path defaults to the canonical in-pod mount.

Failure modes

Symptom (startup log)CauseOperator action
Vault KEK source: failed to resolve KEK after 60s retry window. Path: secret/stellaops/kekVault unreachable or auth refused for entire windowVerify Vault address + network reachability; check auth credentials. Service WILL NOT start until resolved.
Vault read failed: 404 NotFound. Path: secret/data/stellaops/kekPath typo or mount not enabledFix Crypto:Kek:Vault:Path / Mount; verify with vault kv get secret/stellaops/kek.
Vault read failed: 403 Forbidden. Path: secret/data/stellaops/kekAuth principal lacks read capabilityRe-issue the read policy; check vault token capabilities for the principal.
Vault auth mode=approle requires Crypto:Kek:Vault:Auth:RoleId and SecretId.Missing AppRole configSet both values from step 3 above.
Vault read returned no field 'kek'. Path: secret/data/stellaops/kek. Available fields: ...The field name is wrong (secret exists but uses a different key)Set Crypto:Kek:Vault:Field to one of the listed available fields, or re-write the secret with the expected field.
Vault read field 'kek' is empty.Field exists but its value is an empty stringRe-write the secret with non-empty KEK material via vault kv put.
Vault KEK source: secret value is not valid base64.Encoding=base64 set but stored value is not valid base64Either change Crypto:Kek:Vault:Encoding or re-write the secret matching the configured encoding (utf8 / base64 / hex).
Vault auth mode=token requires Crypto:Kek:Vault:Auth:Token to be set.Auth:Mode=token selected with no tokenSet Crypto:Kek:Vault:Auth:Token, or switch to approle / kubernetes.
Vault auth mode=kubernetes: service account token not found at '...'.Auth:Mode=kubernetes but the SA token JWT is missing at the pathVerify the pod mounts a service account token, or set Crypto:Kek:Vault:Auth:KubernetesTokenPath.
Vault auth mode=kubernetes requires Crypto:Kek:Vault:Auth:KubernetesRole.Auth:Mode=kubernetes selected without a configured Vault roleSet Crypto:Kek:Vault:Auth:KubernetesRole to the Vault Kubernetes-auth role name.
Vault login failed: HTTP <code> <reason>. Path: v1/auth/<mode>/login. Verify auth-mode credentials.AppRole / Kubernetes login rejected (bad RoleId/SecretId, wrong K8s role, etc.)Verify the auth-mode credentials; confirm the role exists and the principal is bound to the read policy.
Vault read failed: transport error (...). Path: secret/data/stellaops/kek.Network / TLS failure reaching Vault (treated as transient and retried within the window)Verify Vault address + network reachability + TLS trust; if it persists past StartupRetryWindow the source fails-startup.

The source fails-startup on any persistent failure rather than silently falling back to env-var. This is deliberate: operators who selected Vault selected it for a reason; a quiet downgrade would mask their compliance posture. To roll back to env-var as a recovery move, explicitly set Crypto:Kek:Source=env and restart.

Rotation

KEK rotation while running on Vault follows the standard stella crypto kek rotate flow (RP-028-006):

  1. Update the Vault secret with the new KEK material: vault kv put secret/stellaops/kek kek="$(openssl rand -base64 32)"
  2. Run stella crypto kek rotate --kek-id <id> --dry-run to preview.
  3. Run stella crypto kek rotate --kek-id <id> --confirm to execute.
  4. The walker rolls the per-row kek_version forward; the OLD KEK stays in process memory during the grace window so partial completion is recoverable.

VaultKekSource exposes an InvalidateCache() method that drops the in-process KEK cache so the next ResolveAsync re-fetches fresh material from Vault. Not yet auto-invoked: as of Phase 1, nothing in the rotation control plane (CryptoKekControlPlaneService, the re-seal walker) calls InvalidateCache() on the active IKekSource — the only caller today is the unit test. The “rotate the Vault secret then run kek rotate” flow above therefore relies on the OLD KEK staying loaded for the grace window and the NEW version being introduced by the rotation pathway, not on the source re-reading Vault mid-process. When Phase 2 wires Vault into the services, the rotation pathway should call InvalidateCache() so a fresh resolve picks up the rotated secret; until then, a process restart is the reliable way to re-read a changed Vault secret.

Audit & sentinel guarantees