Runbook: Attestor - HSM Connection Issues

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-005 - Attestor Runbooks

Source-reconciled (2026-05-30; re-verified 2026-05-31): This runbook was rewritten against src/Cryptography/StellaOps.Cryptography.Plugin.Hsm/HsmPlugin.cs, the Doctor check src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Crypto/Checks/HsmPkcs11AvailabilityCheck.cs, the CLI surface in src/Cli/StellaOps.Cli/Commands/CryptoCommandGroup.cs (plus the key group in Commands/Proof/KeyRotationCommandGroup.cs and CommandFactory.cs), and the scope catalog src/Authority/.../StellaOpsScopes.cs. A previous revision documented a stella crypto hsm ... command family, a software-signing fallback, and HSM failover that do not exist in the code; those were removed. The 2026-05-31 re-verification additionally corrected the stella key verify usage (two positional args, and the verb is a stub today), flagged that the Doctor check’s slot/token stages are stubbed to always pass, and noted the --hsm-pin remediation hint is also a non-existent verb. See Decisions & Risks at the end.

Audience: platform and security on-call engineers diagnosing failed HSM-backed signing in a Stella Ops deployment. Use this runbook when attestations stop signing, the Attestor/Signer host fails to start, or the check.crypto.hsm Doctor check reports Fail/Warn. The HSM provider is fail-closed by design — recovery means fixing configuration or connectivity and restarting, never downgrading to software keys.

Metadata

FieldValue
ComponentAttestor / Cryptography
SeverityCritical
On-call scopePlatform team, Security team
Last updated2026-05-30
Doctor checkcheck.crypto.hsm

Symptoms

Note on alerts: No AttestorHsmConnectionFailed Prometheus/alert rule is defined in the repository today. If your deployment ships such an alert, wire it to the HSM plugin HealthCheckAsync result (Unhealthy/Degraded) and record the rule definition alongside this runbook. Treat the alert name here as illustrative, not authoritative.

Impact

Impact TypeDescription
User-facingNo attestations can be signed; releases blocked
Data integrityKeys remain in the HSM; operations resume when the connection is restored
SLA impactAll HSM-backed signing operations blocked; compliance posture at risk

Fail-closed by design. The HSM provider does not fall back to local / software keys when the HSM is unreachable or misconfigured. Outside explicit simulation mode (Development/Testing only), initialization throws before the PKCS#11 library is loaded if any required selector is missing. Treat a missing or failed HSM as a deployment defect to fix, not a condition to work around by switching signing modes. See docs/modules/cryptography/hsm-pkcs11-deployment-health.md (“Fail-Closed Startup Contract” and “Operational Notes”).

Diagnosis

Crypto/HSM diagnostics and configuration changes require an operator token with the appropriate scope (see src/Authority/.../StellaOpsScopes.cs): crypto:read for read-only status, crypto:admin / crypto:profile:admin for provider/profile changes, and crypto:kek:read / crypto:kek:rotate for KEK lifecycle operations.

Quick checks

  1. Run the Doctor HSM check (this is the canonical health gate; the check ID is check.crypto.hsm):

    stella doctor --check check.crypto.hsm

    The check only runs when HSM is enabled (Crypto:Hsm:Enabled=true or Cryptography:Pkcs11:Enabled=true). It reports four stages — module path configured, module file exists, slot access, token presence — but only the first two are real validations today: HsmPkcs11AvailabilityCheck reads Crypto:Hsm:ModulePath (or Cryptography:Pkcs11:ModulePath) and calls File.Exists on it. The slot-enumeration and token-presence stages (CheckSlotsAsync/CheckTokenAsync) are stubbed to always succeed in the current code — they do not open the PKCS#11 module or contact the token. So a Pass from this check confirms only that the module path is set and the file is present, not that the slot/token are actually reachable. Use the HSM plugin’s HealthCheckAsync result (below) for true connectivity. (Flagged for follow-up; see Decisions & Risks.)

  2. Show crypto plugin status (lists the PKCS#11/HSM provider and its health):

    stella crypto plugins status

  3. Show the active crypto provider / FIPS / compliance settings:

    stella crypto status

Deep diagnosis

  1. Inspect the active profile and run provider diagnostics:

    stella crypto profiles show --details --test

    --test runs provider connectivity/diagnostics; --details prints provider capabilities.

  2. Re-run the Doctor check with verbose progress and JSON output to capture the evidence block:

    stella doctor --check check.crypto.hsm --format json --verbose

    On the real failure paths (module path unset / file missing) the evidence carries ModulePath, ConfiguredPath, and Exists. On the Pass path the evidence reports SlotId, SlotLabel, TokenPresent, and TokenLabel — but note these slot/token values come from the stubbed CheckSlotsAsync/ CheckTokenAsync (constant SlotId=0, SlotLabel="SoftHSM slot 0", TokenLabel="stellaops"), so they are placeholders, not live HSM readings. The SlotAccess=failed / Error evidence fields only render if the (stubbed) slot check is ever made to fail; today it cannot. Treat the plugin’s HealthCheckAsync output as the authoritative connectivity signal.

  3. Check the service logs for the Attestor / Signer host. The HSM plugin logs on initialization and per-operation, e.g.:

    • HSM provider connected in {ProviderMode} mode to slot {SlotId} (success)
    • HSM not connected (health probe failed)
    • HSM key id '<id>' is not declared in configured key identifiers (allow-list rejection) Use your standard log access path for the service (there is no stella crypto hsm logs command).

  4. Interpret the health metadata emitted by HealthCheckAsync. Real PKCS#11 mode reports:

    DetailMeaning
    providerMode=pkcs11Provider backed by a real PKCS#11 library
    pkcs11=trueHealth probe used the real PKCS#11 client
    simulation=false / testOnly=falseNot the local test harness
    slotConfigured slot ID
    tokenLabelExpected token label
    configuredKeyCountNumber of declared key identifiers

    If you instead see providerMode=simulation, simulation=true, testOnly=true, or library=in-process-simulator, the service is running the local simulation harness, which is test-only and must never back release evidence.

Resolution

There is no runtime “switch to software signing”, “reconnect”, or “failover to backup HSM” command in Stella Ops today. The provider fails closed and recovers only by fixing the underlying configuration / connectivity and restarting the affected service so the plugin re-initializes.

Immediate mitigation

  1. Confirm the failure class with stella doctor --check check.crypto.hsm (module not configured vs. module missing vs. slot/token failure). This tells you which root-cause section below applies.

  2. Do not bypass the HSM. Switching to local/software keys is not a supported mitigation and contradicts the fail-closed contract — it would invalidate the signing route as release evidence. If signing must continue, restore the HSM route, do not downgrade it.

  3. After fixing the configuration or connectivity, restart the affected service (Attestor / Signer host) so InitializeCryptoServiceAsync re-runs and re-opens the slot/session. The plugin connects on startup; there is no live reconnect verb.

Root cause fix

If the PKCS#11 module path is not configured or the file is missing (Doctor check Fail):

  1. Configure the module / library path in the service configuration. The Doctor check reads Crypto:Hsm:ModulePath (or Cryptography:Pkcs11:ModulePath), while the HSM plugin itself binds Cryptography:Hsm:LibraryPath:

    Cryptography:
  Hsm:
    LibraryPath: "/usr/lib/softhsm/libsofthsm2.so"   # Windows: C:\SoftHSM2\lib\softhsm2.dll

    The Doctor check output prints a stella crypto config set --hsm-module … remediation hint, but that CLI verb is not implemented today — there is no config subcommand under stella crypto. Apply the path through the service configuration (env var / config file / Platform environment settings) instead, and restart the service. (Flagged for follow-up; see Decisions & Risks.)

  2. Verify the library architecture matches the service image (an x64-only .so cannot load in an ARM64 .NET process). See the “ARM64 Runtime Contract” in docs/modules/cryptography/hsm-pkcs11-deployment-health.md.

If startup fails closed with a “requires …” error: the plugin requires LibraryPath, SlotId, TokenLabel, Pin, and at least one key identifier (KeyIds or operation-specific IDs) outside simulation mode. Supply the missing selectors via the service configuration (PIN must come from a secret source, not source-controlled config):

Cryptography:
  Hsm:
    LibraryPath: "/usr/lib/softhsm/libsofthsm2.so"
    SlotId: 0
    TokenLabel: "StellaOps-Prod"
    Pin: "${HSM_PIN}"
    KeyIds: "stellaops-signing-2026,stellaops-encryption-2026"

If slot access fails (Doctor Fail, “PKCS#11 slot access failed”):

  1. Check module permissions: ls -la <module-path>.
  2. Initialize the slot/token if needed (SoftHSM example):
    softhsm2-util --init-token --slot 0 --label "StellaOps-Prod"

If the token is not accessible (Doctor Warn, “PKCS#11 token not accessible”):

  1. Initialize the token and configure the PIN (the plugin validates the configured TokenLabel against the slot — a mismatch fails startup):
    softhsm2-util --init-token --slot 0 --label StellaOps-Prod --pin <pin> --so-pin <so-pin>
  2. Provide the user PIN through your secret source (Cryptography:Hsm:Pin).

If signing fails with an allow-list error (HSM key id '<id>' is not declared in configured key identifiers): the runtime KeyId is not in the configured allow-list. Add it to KeyIds (or the relevant SigningKeyIds / VerificationKeyIds / EncryptionKeyIds / DecryptionKeyIds) and restart the service.

Verification

# Re-run the HSM Doctor check; expect Pass (confirms module path set + file
# present — slot/token stages are stubbed, so this is necessary, not sufficient)
stella doctor --check check.crypto.hsm --format json

# Confirm the crypto plugin reports the HSM/PKCS#11 provider healthy
stella crypto plugins status

# Confirm the active profile/provider and run provider diagnostics
stella crypto profiles show --details --test

Do not rely on stella key verify as an HSM verification step today. The top-level stella key verify command (the KeyRotationCommandGroup tree: key list|add|revoke|rotate|status|history|verify) takes two positional arguments — <anchorId> (a GUID trust-anchor id) and <keyId> — plus an optional --signed-at <iso8601>, i.e. stella key verify <anchorId> <keyId>. More importantly, every verb in that group is currently a stub: it prints Error: key rotation verify requires a configured key rotation service. No simulated trust-anchor change or validity result was produced. and returns a system-error exit code. It does not contact the HSM and cannot confirm a signing key’s validity in the current build. (Flagged; see Decisions & Risks.)

Prevention

Decisions & Risks