Runbook: Regional Crypto Operations

Audience: Platform and Security on-call engineers operating the Stella Ops cryptographic subsystem — regional compliance profiles, KEK rotation, HSM-backed signing, eIDAS provider packs, and certificate-chain validation across the release control plane.

Sprint: SPRINT_20260117_029_Runbook_coverage_expansion Task: RUN-002 - Crypto Subsystem Runbook Status: Production-ready (last reconciled against src/ 2026-05-31 UTC)

Scope

Cryptographic subsystem operations: regional crypto compliance-profile configuration, master KEK rotation, PKCS#11 HSM-backed signing, eIDAS provider-pack intake, and certificate-chain validation. The built-in compliance profiles are world (default), fips, eidas, gost, sm, and kcmvp (see src/__Libraries/StellaOps.Cryptography/ComplianceProfiles.cs).

Reconciliation note: this runbook was originally written around an aspirational stella crypto {hsm,certs,audit,diagnostics} CLI and several metrics/alerts that do not exist in src/. It has been reconciled against the actual CLI (src/Cli/StellaOps.Cli/Commands/CryptoCommandGroup.cs, DoctorCommandGroup.cs), the crypto core (src/__Libraries/StellaOps.Cryptography/), the crypto plugins (src/Cryptography/), the Doctor crypto checks (src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Crypto/), and the scope catalog (StellaOpsScopes.cs). Sections describing unimplemented surfaces are annotated inline.

Pre-flight Checklist

Environment Verification

# Run the embedded Doctor crypto checks (category name is "Crypto"; matching is case-insensitive)
stella doctor --category crypto

# Verify the active tenant crypto compliance profile
stella crypto profile show

# Show crypto + compliance installation settings (provider region, FIPS mode)
stella crypto status

# List the available crypto profiles / plugin status
stella crypto profiles list
stella crypto plugins status

# List configured signing keys
stella crypto keys list

Reconciliation note (verified against src/Cli/StellaOps.Cli/Commands/CryptoCommandGroup.cs): the stella crypto group exposes sign, verify, provider set|show, algorithm set, fips enable|disable, status, profile set|show|validate, profiles list|select|show, plugins status, keys generate|list|import|export, encrypt, decrypt, hash, and the kek subgroup. There is no stella crypto providers list or stella crypto keys status subcommand. stella crypto keys list/generate/import/export currently emit illustrative placeholder output (Sprint SPRINT_20260118_014 / CLI-E-004) rather than reading live key state.

Metrics to Watch

The crypto core (StellaOps.Cryptography) publishes its meter under the name stellaops.crypto (see src/__Libraries/StellaOps.Cryptography/CryptoProviderMetrics.cs):

Reconciliation note: the previously-documented metrics stella_crypto_operations_total, stella_crypto_operation_duration_seconds, stella_hsm_availability, and stella_cert_expiry_days are NOT IMPLEMENTED — no such instruments exist anywhere under src/. Operation-latency, HSM-availability, and certificate-expiry instrumentation are roadmap items; do not build alerts on those names.

Regional Crypto Profiles

The built-in compliance profiles are defined in src/__Libraries/StellaOps.Cryptography/ComplianceProfiles.cs. Each profile maps hashing/HMAC purposes (graph, symbol, content, merkle, attestation, interop, secret) onto a concrete algorithm set; the values below are the load-bearing facts from that file.

Profile IDUse CaseHash / HMAC algorithms (per ComplianceProfiles)Standard name
worldDefault, most deploymentsBLAKE3-256 for graph content-addressing, SHA-256 for symbol/content/merkle/attestation/interop, Argon2id for secrets, HMAC-SHA256ISO/Default
fipsUS Federal / FedRAMPSHA-256 everywhere (BLAKE3 is not FIPS-approved), PBKDF2-SHA256 for secrets, HMAC-SHA256FIPS 140-3
eidasEuropean UnionSHA-256 (per ETSI TS 119 312), Argon2id for secrets, HMAC-SHA256eIDAS / ETSI TS 119 312
gostRussian FederationGOST R 34.11-2012 (Stribog-256) hashing, SHA-256 for interop, Argon2id for secrets, HMAC-GOST3411GOST R 34.11-2012
smChinaSM3 hashing, SHA-256 for interop, Argon2id for secrets, HMAC-SM3GB/T 32905-2016 (SM3)
kcmvpRepublic of KoreaSHA-256 everywhere, Argon2id for secrets, HMAC-SHA256KCMVP (Korea)

Reconciliation notes:

  • The canonical default profile ID is world, not international. international is only a provider-region alias accepted by stella crypto provider set (alongside eidas/russian/chinese); it is not a ComplianceProfile ID.
  • Two different profile-selection surfaces accept different value sets (verified against CryptoCommandGroup.cs):
    • stella crypto profile set <profile> (the tenant-compliance verb wired to the Platform tenant-compliance endpoint) normalizes via TenantProfileAliases and accepts world, fips, gost, sm, kcmvp, eidas (plus aliases international/default → world, eu/europe → eidas, russian/ru → gost, chinese/cn → sm, korean/kr → kcmvp). This is the surface the procedures below use.
    • stella crypto profiles select <profile> (the local/hybrid preference-store verb) is backed by the in-CLI GetAvailableCryptoProfiles() catalog and accepts only international, fips, eidas, gost, sm. It does not know world or kcmvp, and it uses international (not world). Prefer profile set for tenant compliance; profiles select only persists a local preference (~/.stellaops/appsettings.crypto.yaml) or the platform admin endpoint.
  • fips is declared as FIPS 140-3 in ComplianceProfiles.cs; note the Doctor check.crypto.fips plugin and parts of this runbook still describe it as FIPS 140-2. Treat the 140-3 designation in the profile definition as authoritative.
  • The table previously omitted kcmvp (Korea), which is a fully-defined built-in profile.
  • The previous “Key Algorithms” column listed signature primitives (RSA-PSS, ECDSA, Ed25519, SM2, GOST R 34.10) that are governed by the signing providers/plugins, not by the ComplianceProfile hash/HMAC mapping. They are retained conceptually but the table above reflects what the profile object actually configures.

Switching Profiles

  1. Pre-switch verification:

    # List available profiles, then validate the currently-active one
# (plugin reachable / FIPS-mode disagreement). `validate` is wired against the
# Platform admin endpoint GET /api/v1/admin/crypto/profile/validate.
stella crypto profiles list
stella crypto profile validate

  2. Profile switch:

    # Set the tenant crypto compliance profile (positional argument, NOT --profile).
# Valid values: world | fips | gost | sm | kcmvp | eidas
stella crypto profile set <target-profile> --tenant <tenant-id>

# Optionally re-seal existing tenant rows to the new algorithm in one step
# (drives a KEK rotation; see SP-005). Mapping: fips/eidas/world/kcmvp -> AES-256-GCM,
# sm -> SM4-GCM, gost -> GOST-GCM.
stella crypto profile set <target-profile> --reseal

    Restart the affected services through your normal compose/host orchestration after a profile change so plugins reload.

  3. Post-switch verification:

    # `--check` takes a SINGLE check ID; run one per profile of interest.
stella doctor --check check.crypto.fips
stella doctor --check check.crypto.eidas
stella doctor --check check.crypto.gost
stella doctor --check check.crypto.sm

# Or run the whole Crypto category at once:
stella doctor --category crypto

Reconciliation notes:

  • stella crypto profile verify and stella crypto audit do not exist. The real verb is stella crypto profile validate; there is no signature-compatibility audit command.
  • stella crypto profile set takes the profile as a positional argument with optional --tenant, --reseal, and --reseal-kek-id flags — not a --profile option.
  • There is no stella service restart command in the CLI; service lifecycle is handled by the deployment orchestrator (compose/host), not by stella.
  • stella doctor --check accepts a single check ID; comma-separated lists are not parsed. There is no check.crypto.kcmvp Doctor check (the available crypto checks are check.crypto.fips, check.crypto.eidas, check.crypto.gost, check.crypto.sm, check.crypto.hsm, and check.crypto.certchain).

Standard Procedures

SP-001: Signing Key Generation

Frequency: Per policy / on key provisioning Duration: ~15 minutes (no downtime)

  1. Generate a new key pair:

    # Algorithm: rsa | ecdsa | ed25519 (default ecdsa). --output/-o is the path prefix
# (writes <prefix>.key and <prefix>.pub). --size/-s applies to RSA; --password encrypts the private key.
stella crypto keys generate --algorithm ecdsa --output signing-$(date +%Y%m)

  2. Confirm the key is listed and exercise the signer:

    stella crypto keys list

# `sign` requires a file via --input (it does not read stdin). --provider can override the
# active provider (e.g. gost-cryptopro, eidas-tsp, sm-remote); --format is dsse|jws|raw.
stella crypto sign --input ./artifact.bin --output ./artifact.bin.sig
stella crypto verify --input ./artifact.bin --signature ./artifact.bin.sig

Reconciliation notes (verified against CryptoCommandGroup.cs):

  • stella crypto keys generate accepts --algorithm/-a, --size/-s, --output/-o, and --password. There is no --type, --name, or --provider option, and there are no keys activate, keys revoke, keys block, or keys schedule-deactivation subcommands.
  • The keys generate|list|import|export handlers currently print illustrative placeholder output (Sprint SPRINT_20260118_014 / CLI-E-004); they are a forward-looking surface, not a live key-store manager. The implemented, live key-lifecycle control plane is KEK rotation (see SP-005).
  • stella crypto sign requires --input <path>; the echo ... | stella crypto sign stdin pattern is not supported.

SP-002: Certificate Renewal

When: Certificate expiring within 30 days

NOT IMPLEMENTED (roadmap). There is no stella crypto certs command group in the CLI (stella crypto certs check-expiry|csr|install|generate-self-signed do not exist), and there is no stella service restart. The only certificate-related capability that exists today is the Doctor check check.crypto.certchain (CertChainValidationCheck), which validates an already-installed chain. Until a certificate-management surface lands, perform CSR generation, issuance, and installation with your platform’s TLS tooling (e.g. OpenSSL / your reverse proxy / orchestrator config), then verify with the Doctor check below.

  1. Verify the installed certificate chain:

    stella doctor --check check.crypto.certchain

  2. CSR generation, issuance, and installation: use external TLS tooling (out of stella scope today).

  3. After installing a renewed certificate, restart the affected services through your deployment orchestrator (compose/host), then re-run the chain check above.

SP-003: HSM Health Check

Frequency: Daily (automated) or on-demand

NOT IMPLEMENTED (roadmap) for the CLI surface. There is no stella crypto hsm command group (hsm status, hsm slots list, hsm test-sign, hsm test-module, hsm sessions release do not exist). The HSM integration that does exist is the PKCS#11 signing plugin StellaOps.Cryptography.Plugin.Hsm (HsmPlugin / Pkcs11HsmClientImpl), which provides HSM-backed signing/AEAD (HSM-RSA-*, HSM-ECDSA-P256/P384, HSM-AES-128/256-GCM) via a configured PKCS#11 LibraryPath. There is also a Doctor check check.crypto.hsm (HsmPkcs11AvailabilityCheck).

  1. Run the Doctor PKCS#11 availability check:

    stella doctor --check check.crypto.hsm

  2. Slot access, test-sign, session management, and vendor health metrics (free objects/sessions, temperature) are vendor-tool / PKCS#11-module responsibilities today, not stella subcommands.

SP-004: eIDAS Provider-Pack Intake

When: A deployment requires eIDAS BaselineT, BaselineLT, or BaselineLTA release evidence.

  1. Select the source:

    • QTSP/QSCD-backed provider pack from the selected qualified trust provider or bridge integrator.
    • Compliance-owned sealed evidence pack approved for offline validation.

  2. Confirm the required contents:

    • RFC 3161 timestamp tokens.
    • OCSP responses or CRLs.
    • Archive timestamp tokens when BaselineLTA is required.
    • Signing certificate chain and TSA certificate chain.
    • TSL and LOTL snapshots.
    • Accepted/rejected fixture verdicts, expiry metadata, and SHA-256 hashes.

  3. Complete legal and license review before copying bytes into a repository, offline kit, runtime-assets bundle, or CI asset store.

  4. Record the intake in the deployment evidence register:

    • packId
    • source owner
    • allowed storage location
    • local sealed path
    • manifest SHA-256
    • jurisdiction and evidence profile
    • expiry date
    • legal/license disposition

  5. Point the eIDAS plugin at the local sealed pack. The plugin (EidasPlugin) reads, in order of precedence: Eidas:TrustedList:OfflineTrustMaterialPackPath (the primary key) and then Eidas:TrustedList:FixturePackManifestPath (fallback). Related keys include Eidas:TrustedList:QtspAgreementPath, Eidas:TrustedList:AllowHistoricalPreTlv6Use, and Eidas:TrustedList:IsHistoricalReplay. When RequireProductionClaimAlgorithms is true and the offline pack path is unset, the plugin refuses startup (fail-closed).

    Then validate the active profile and run the Doctor eIDAS check:

    stella crypto profile validate
stella doctor --check check.crypto.eidas

Reconciliation note: there is no stella crypto diagnostics command. Use stella crypto profile validate (wired to GET /api/v1/admin/crypto/profile/validate) and the Doctor crypto checks (stella doctor --check check.crypto.eidas / --category crypto, exportable via stella doctor export) to capture eIDAS-readiness evidence. Also prefer the primary config key Eidas:TrustedList:OfflineTrustMaterialPackPath over the FixturePackManifestPath fallback.

Until a selected pack passes offline preflight and later cryptographic validation, BaselineT, BaselineLT, and BaselineLTA remain fail-closed. Do not substitute the universal simulator, synthetic timestamp bytes, or live TSL/LOTL fetches for provider-pack evidence.

SP-005: Master KEK Rotation

When: Scheduled KEK rotation, suspected KEK compromise, or after a profile change that requires re-sealing tenant rows to a new algorithm.

The KEK lifecycle is the implemented, live key-rotation control plane (Sprint SPRINT_20260529_028). All verbs route through IPlatformCryptoKekClient to the Platform admin endpoints under /api/v1/admin/crypto/kek, gated by the crypto:kek:read / crypto:kek:rotate scopes.

  1. Inspect current KEK state and history:

    # Default KEK id is platform:connector-credentials:v1
stella crypto kek status --kek-id platform:connector-credentials:v1
stella crypto kek history --kek-id platform:connector-credentials:v1

  2. Plan the rotation (no mutation):

    stella crypto kek rotate --kek-id platform:connector-credentials:v1 --dry-run

  3. Execute and stream progress until completion / abort / hard fail:

    stella crypto kek rotate --kek-id platform:connector-credentials:v1 --confirm

  4. If a rotation must be stopped mid-flight:

    stella crypto kek abort --kek-id platform:connector-credentials:v1 --rotation-id <guid>

Note: the rotate options --new-source (env|file|vault|hsm), --new-source-ref, and --grace-hours are accepted by the CLI today and validated locally, but full server-side plumbing is staged (Phase 2 / RP-028-008). Per the on-prem-first posture, KEK sources are Vault / file / env / HSM — never a cloud-managed KMS.

Incident Procedures

Alert-name reconciliation: the alert identifiers below (StellaHsmUnavailable, StellaCertExpired, StellaFipsNotEnabled) appear only in this runbook — no Prometheus/alerting rule defines them. Treat them as proposed alert names, not live ones, until the corresponding rules are authored against real instruments (today the only crypto metrics are crypto_provider_resolutions_total and crypto_provider_resolution_failures_total).

INC-001: HSM Unavailable

Symptoms:

Investigation:

# PKCS#11 availability via Doctor (no `stella crypto hsm` command exists)
stella doctor --check check.crypto.hsm

# Test the configured PKCS#11 library / network path with vendor tooling (e.g. pkcs11-tool),
# since `stella network test` does not exist.

Resolution:

  1. Network issue:

    • Verify network path to HSM
    • Check firewall rules
    • Verify HSM appliance is powered on

  2. Session exhaustion / module reload:

    • Release stale sessions and reload the PKCS#11 module via the HSM vendor’s tooling.
    • Restart the signing service through your deployment orchestrator (compose/host); there is no stella service restart command.

  3. HSM failure:

    • Fail over to secondary HSM (if configured)
    • Contact HSM vendor support
    • Consider temporary fallback to software keys (with approval)

Reconciliation note: there is no stella crypto hsm command group and no stella network test. The HSM integration is the PKCS#11 signing plugin StellaOps.Cryptography.Plugin.Hsm; operational visibility is the Doctor check check.crypto.hsm plus vendor PKCS#11 tools.

INC-002: Signing Key Compromised

CRITICAL - Follow incident response procedure

Reconciliation note: there are no stella crypto keys revoke|block|activate or stella crypto audit export subcommands. The live, implemented rotation path for compromise response is KEK rotation (SP-005) plus signer re-provisioning through your secret backend (Vault/file/HSM). The keys subcommands that exist (generate|list|import|export) emit placeholder output and are not a live revocation control plane.

  1. Immediate containment:

    • Remove/disable the compromised key material at the secret backend (Vault/file/HSM), and rotate the master KEK if the KEK itself is implicated:
      stella crypto kek rotate --kek-id <kek-id> --confirm

  2. Provision a replacement signing key through your secret backend / provider configuration, then verify with stella crypto sign --input ... / stella crypto verify ....

  3. Notify downstream:

    • Update trust registries with the new key
    • Notify relying parties
    • Publish key revocation notice

  4. Forensics:

    • Collect Authority audit logs (authority.audit.read scope) and Doctor exports (stella doctor export); there is no dedicated stella crypto audit export command.

INC-003: Certificate Expired

Symptoms:

Immediate Resolution:

Reconciliation note: stella crypto certs install|generate-self-signed and stella service restart do not exist. Certificate install / emergency self-signed generation is done with external TLS tooling (OpenSSL / reverse proxy / orchestrator config); stella only validates an installed chain.

  1. If a renewed certificate is available, install it via your platform’s TLS tooling, restart the affected services through the deployment orchestrator, then verify:

    stella doctor --check check.crypto.certchain

  2. If renewal is not ready, generate an emergency self-signed certificate with OpenSSL (NOT for production use), install it the same way, and re-run the chain check above.

  3. Expedite the certificate renewal process.

INC-004: FIPS Mode Not Enabled

Symptoms:

Resolution:

  1. Linux:

    # Enable FIPS mode
sudo fips-mode-setup --enable

# Reboot required
sudo reboot

# Verify after reboot
fips-mode-setup --check

  2. Windows:

    • Enable via Group Policy
    • Or via registry:
      Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\Lsa\FipsAlgorithmPolicy" -Name "Enabled" -Value 1
Restart-Computer

  3. Restart Stella services through your deployment orchestrator (compose/host), then re-run the Doctor FIPS check:

    stella doctor --check check.crypto.fips

Reconciliation notes: the Linux (fips-mode-setup --enable) and Windows (Group Policy / the FipsAlgorithmPolicy registry key) steps above match the remediation emitted by the Doctor FipsComplianceCheck plugin (src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Crypto/Checks/FipsComplianceCheck.cs). The check.crypto.fips check only runs (CanRun) when the configured crypto profile is a FIPS/FedRAMP/us-gov profile — set the profile to fips first (see Switching Profiles) so the check is active. Note the profile object names this “FIPS 140-3” while the Doctor plugin labels it “FIPS 140-2”.

Regional-Specific Procedures

GOST Configuration (Russian Federation)

  1. Install the GOST OpenSSL engine on the host (package name varies by distro/OpenSSL version, e.g. libengine-gost-openssl1.1 on Debian/Ubuntu).

  2. Set the tenant compliance profile to GOST:

    # Positional argument; NOT --profile.
stella crypto profile set gost

    GOST engine paths are configured through the GOST plugin / host OpenSSL configuration, not via a stella crypto config set command (which does not exist).

  3. Verify:

    stella doctor --check check.crypto.gost

SM Configuration (China)

  1. Ensure OpenSSL 1.1.1+ with SM support:

    openssl version
openssl list -cipher-algorithms | grep -i sm

  2. Set the tenant compliance profile to SM:

    stella crypto profile set sm

  3. Verify:

    stella doctor --check check.crypto.sm

Reconciliation note: stella crypto profile set takes the profile as a positional argument, not --profile. There is no stella crypto config set command; engine/library paths are set via plugin/host configuration (e.g. the GOST plugin and host OpenSSL config). For SM, a remote SM service is also available via the SmRemote module (scopes sm-remote:hash|encrypt|decrypt|sign|verify).

Monitoring Dashboard

Reconciliation note: a dedicated “Crypto Subsystem” Grafana dashboard is not confirmed in the repo, and most panels below depend on instruments that do not yet exist (signing latency, key usage by key ID, HSM availability, certificate-expiry countdown). The only crypto instruments emitted today are crypto_provider_resolutions_total and crypto_provider_resolution_failures_total (meter stellaops.crypto). Treat the panel list below as a target dashboard design, not a shipped one.

Proposed panels:

Evidence Capture

Reconciliation note: stella crypto diagnostics does not exist. Capture crypto evidence with the implemented surfaces below.

# Crypto/compliance installation settings + active profile
stella crypto status
stella crypto profile show
stella crypto profiles list
stella crypto plugins status

# Validate the active profile (Platform admin endpoint)
stella crypto profile validate --format json

# Run and export the Doctor crypto checks as an evidence bundle
stella doctor --category crypto --format json
stella doctor export

# KEK lifecycle evidence
stella crypto kek status --kek-id platform:connector-credentials:v1 --format json
stella crypto kek history --kek-id platform:connector-credentials:v1 --format json

Captured evidence covers:

Escalation Path

  1. L1 (On-call): Doctor crypto checks (stella doctor --category crypto), profile status/validation
  2. L2 (Security team): KEK rotation (SP-005), HSM/PKCS#11 issues, profile switches
  3. L3 (Crypto SME): Algorithm issues, compliance questions, eIDAS provider-pack validation
  4. HSM Vendor: Hardware failures

RBAC: KEK lifecycle endpoints are gated by crypto:kek:read / crypto:kek:rotate; crypto-provider admin surfaces by crypto:read / crypto:admin / crypto:profile:admin; Doctor by doctor:run (and doctor:run:full / doctor:export); forensic audit by authority.audit.read (see StellaOpsScopes.cs).

Last reconciled against src/: 2026-05-31 (UTC). Original authored: 2026-01-17.