Production signing runbook (compliance evidence)

Audience. The release approver / operator cutting a real (non-lab) release who must turn the compliance-evidence signing surfaces from the lab’s clearly-marked dev-stable software seed into a sovereign-provider-provisioned production key, and publish the matching verification key so air-gapped operators verify evidence fully offline.

Claim boundary (read first). The lab signs with a software dev-stable key that is not a secret and is clearly marked as such — it exists only so the golden-path demo produces reproducible signatures. This runbook is operator support: it tells you how to provision real signing. It does not make Stella (or your estate) “NIS2/CRA/DORA compliant” or eIDAS-qualified — the operator remains the regulated party, and “qualified” (QES) is an explicit, separate opt-in (§6).

On-prem first (non-negotiable). The production key backend is a sovereign provider — HashiCorp Vault (flagship KEK/secret backend), an in-process PKCS#11 HSM, or a software sovereign provider plugin (eu.eidas.soft, fips.ecdsa.soft, GOST, SM). A cloud-managed KMS (AWS/Azure/GCP) is never a default and is only ever reachable as an explicit, operator-selected ProviderOverride — the selection routine never picks one for you (SignatureProviderSelection.Resolve returns a non-null ProviderHint only when you pass an explicit override). See MEMORY feedback_on_prem_first_no_cloud_defaults.

This runbook is the production-signing on-ramp referenced from the release checklist. It consolidates the scattered “Production hardening” notes into one authoritative procedure for the two compliance-evidence signing surfaces:

Everything below is a configuration / key-provisioning change. No signer code changes between the lab and production — that is the entire point of the provider-selection routine.

1) How a signer chooses its algorithm + provider

Both surfaces resolve their signing algorithm and provider through one routine:

• src/__Libraries/StellaOps.Cryptography/SignaturePurpose.cs — the well-known purposes (capsule-seal, soa-export, evidence-bundle, verdict-attestation, regulatory-payload, generic-dsse).

• src/__Libraries/StellaOps.Cryptography/SignatureProviderSelection.cs — the routine:

  SignatureProviderSelection.Resolve(
      new SignatureSelectionRequest(
          Purpose:           SignaturePurpose.CapsuleSeal,   // or SoaExport
          ProfileId:         "<active compliance profile>",  // optional
          AlgorithmOverride: "<configured Signing:Algorithm>",// optional
          ProviderOverride:  "<configured Signing:Provider>"))// optional
    => SignatureSelection(AlgorithmId, ProviderHint?)
  

A call site requests a signer by purpose. The routine resolves the canonical algorithm for (profile + purpose) (honouring any explicit override), and the ICryptoProviderRegistry then resolves the concrete provider from the active profile’s preferred order. So swapping the lab software key for a sovereign provider/HSM is a selection/config change, never a code change.

1.1 Three distinct config inputs (do not conflate)

These are three separate keys with three separate jobs (verified in src/):

STELLAOPS_CRYPTO_PROFILE (regional bundle: default/china/russia/eu) is not the same as STELLAOPS_CRYPTO_COMPLIANCE_PROFILE (algorithm family: world/fips/gost/sm/kcmvp/eidas). The regional compose overlay typically sets the compliance profile for you (e.g. the eu overlay), but they are distinct keys — verify both when wiring a region.

1.2 Per-purpose algorithm defaults (behaviour-preserving)

From SignatureProviderSelection.cs (WorldDefaults + the regional switch):

So under the default/world profile the lab seals capsules with Ed25519 and signs the SoA with ES256 — exactly what the live compose sets. Switching to a regional profile changes the algorithm family without touching signer code.

1.3 Provider ids and their gates (verified in src/)

The software sovereign providers are gated off by default and must be explicitly allowed (do not enable a profile you are not certifying). Verified against src/__Libraries/StellaOps.Cryptography/CompliancePolicyCryptoProviders.cs:

These software providers are non-certified (no QSCD / FIPS module enforced). Making any of them “qualified eIDAS” / certified-FIPS is a separate decision (§6).

2) Surface A — EvidenceLocker decision-capsule key

2.1 What the lab does today

The live stack (devops/compose/docker-compose.stella-services.yml, evidence-locker-web and evidence-locker-worker) seals capsules with a stable Ed25519 key on the bouncycastle.ed25519 provider, selected through the routine but seeded from a dev-stable base64 seed in config:

EvidenceLocker__Signing__Enabled: "true"
EvidenceLocker__Signing__UseCryptoRegistry: "true"          # routes through the abstraction
EvidenceLocker__Signing__Algorithm: "ED25519"
EvidenceLocker__Signing__KeyId: "evidence-locker-capsule-key"
EvidenceLocker__Signing__Provider: "bouncycastle.ed25519"
EvidenceLocker__Signing__AllowEphemeralKeyMaterial: "false" # no fresh-key fallback
# DEV-STABLE seed — NOT a secret, lab/golden-path only:
EvidenceLocker__Signing__KeyMaterial__Ed25519PrivateKeyBase64: "${STELLAOPS_EVIDENCELOCKER_CAPSULE_ED25519_SEED_B64:-REDACTED-ROTATED-2026-07-30}"

Config keys verified in EvidenceLockerOptions.cs (SigningOptions / SigningKeyMaterialOptions). The signer is CryptoRegistryCapsuleSigner (src/EvidenceLocker/StellaOps.EvidenceLocker/Capsules/CryptoRegistryCapsuleSigner.cs): it calls SignatureProviderSelection.Resolve(CapsuleSeal, …) with Signing:Algorithm and Signing:Provider as explicit overrides, then asks the registry for the key by KeyId. In the lab the key is provisioned in-process from the stable seed (LoadSigningKeyMaterial); in production the same KeyId is pre-provisioned on the sovereign provider, so ResolveSigner succeeds first and LoadSigningKeyMaterial is never reached.

2.2 Production wiring

1. Keep UseCryptoRegistry=true. The legacy raw-crypto capsule signers (Ed25519CapsuleSigner / EcdsaCapsuleSigner) bypass the abstraction and are reachable only when UseCryptoRegistry=false AND raw key material is configured. Production deployments MUST keep UseCryptoRegistry=true so no compliance signing path bypasses the abstraction (see signing-provider-selection.md §3).
2. Pick the profile + provider for your region (§1.1, §1.3): set the compliance profile, enable the chosen provider’s env gate, and put it first in the registry preferred order. For the eu profile use eu.eidas.soft (EIDAS_SOFT_ALLOWED=1); for fips use fips.ecdsa.soft (FIPS_SOFT_ALLOWED=1); etc.
3. Provision the key on the provider — not raw bytes in config. Pre-provision the evidence-locker-capsule-key KeyId on the chosen provider/HSM (Vault transit or PKCS#11). Point EvidenceLocker:Signing:KeyId at it and (optionally) pin the provider with EvidenceLocker:Signing:Provider. Leave Provider empty to let the registry pick from the profile order.
4. Remove the dev-stable seed and keep the ephemeral fallback off:
   • Delete EvidenceLocker__Signing__KeyMaterial__Ed25519PrivateKeyBase64 (do not ship the lab seed to production).
   • EvidenceLocker__Signing__AllowEphemeralKeyMaterial: "false" — already false in the lab; keep it false. With no inline seed and no ephemeral fallback, the signer requires the key to be pre-provisioned on the provider (it will throw on a missing key rather than silently mint a fresh one that would break offline verify).
   • EvidenceLocker__Signing__AllowUnsignedCapsules: "false" (or omit). The production startup gate refuses unsigned capsules (§2.3).
5. Set the algorithm to match the profile (or leave it pinned). Under a regional profile the routine fills the per-purpose default; if you pin Signing:Algorithm, the registry validates the resolved provider supports it.

2.3 The production startup gate

EvidenceLockerRuntimeConfigurationValidator (verified in …/Infrastructure/Hosting/) fails fast on non-local-harness startup unless the capsule signer has durable key material. ValidateCapsuleSigner passes when any of: UseCryptoRegistry=true, an Ed25519 seed is configured, an EC PEM is configured, or AllowEphemeralKeyMaterial=true. ValidateAllowUnsignedCapsules throws if AllowUnsignedCapsules=true outside the local harness. The local-harness escape (EvidenceLocker:LocalHarness:Enabled=true) only takes effect in Development/Testing (or the TestingLocalHarness environment) — a production host ignores it.

Production gotcha. Because UseCryptoRegistry=true alone satisfies the gate, the validator will not catch a missing provider key — it only guards against the NullCapsuleSigner. The real safety net is AllowEphemeralKeyMaterial=false (no fresh key) plus the published JWKS (§4): if the provider key is absent, the signer throws on sign and the JWKS endpoint returns 503 rather than silently signing with the wrong key. Verify §7 round-trips before you sign release evidence.

3) Surface B — ExportCenter NIS2 SoA export key

3.1 What the lab does today

The live stack signs the NIS2 SoA bundle with a stable ES256 key provisioned on an in-process DefaultCryptoProvider from a dev-stable EC P-256 PEM.

Host and file moved 2026-09-13 (SPRINT_20260722_025 OK-10). These keys are carried by offlinekit-webin devops/compose/docker-compose.offlinekit-consolidated.yml, not by export-web in docker-compose.stella-services.yml — verified at HEAD: nineteen Export__Nis2Soa__* keys in the former, zero in the latter, and the export-web container is stopped. One key also changed shape: the overlay sets Export__Nis2Soa__SigningPrivateKeyPemFile (a mounted path) rather than the inline Export__Nis2Soa__SigningPrivateKeyPem shown below, so the §2.12 half of the hardening step in §7.2 is already taken.

Export__Nis2Soa__AuthorityBaseUrl: "https://stella-ops.local"   # the GATEWAY, not the internal authority alias
Export__Nis2Soa__StorageRoot: "/var/lib/stella/exportcenter/nis2-soa"
Export__Nis2Soa__SigningKeyId: "nis2-soa-export-key"
Export__Nis2Soa__SigningAlgorithmId: "ES256"
Export__Nis2Soa__SigningProviderHint: "default"
Export__Nis2Soa__ExpectedSignerKeyId: "nis2-soa-export-key"
Export__Nis2Soa__SigningPrivateKeyPem: |                        # DEV-STABLE — lab only
  -----BEGIN PRIVATE KEY-----
  …
  -----END PRIVATE KEY-----
Export__Nis2Soa__TrustRootMode: "AirGap"
Export__Nis2Soa__TrustRoots__0__TrustRootId: "stellaops-nis2-soa-dev-stable-2026"
Export__Nis2Soa__TrustRoots__0__KeyId: "nis2-soa-export-key"
Export__Nis2Soa__TrustRoots__0__PublicKeyPem: |
  -----BEGIN PUBLIC KEY-----
  …
  -----END PUBLIC KEY-----
Export__Nis2Soa__TrustRoots__0__Fingerprint: "sha256:8940…5459"
Export__Nis2Soa__TrustRoots__0__NotBeforeUtc: "2026-01-01T00:00:00Z"
Export__Nis2Soa__TrustRoots__0__NotAfterUtc: "2027-01-01T00:00:00Z"
Export__Nis2Soa__TrustRoots__0__SignerProfiles__0: "tenant-regulatory-export"

Config keys verified in Nis2SoaExportOptions.cs. The live signed run is POST /v1/exports/nis2/soa/runs (required scope export:operate, verified in ExportSurfacingClient.cs / ExportSurfacingClientTests.cs). The registry + key provisioning happens in AddNis2SoaExportServices (Nis2SoaExportServiceCollectionExtensions.cs): it provisions the SigningKeyId from SigningPrivateKeyPem only when no ICryptoProviderRegistry is already registered — i.e. the inline PEM is purely the lab/dev-stable path.

3.2 Production wiring

1. Pre-register a sovereign-provider registry. AddNis2SoaExportServices explicitly skips provisioning from the PEM when an ICryptoProviderRegistry is already registered (verified: if (services.Any(… == typeof(ICryptoProviderRegistry))) return;). In production register your sovereign-provider registry (Vault / PKCS#11 HSM / eu.eidas.soft / fips.ecdsa.soft) so it owns the key.
2. Provision the nis2-soa-export-key KeyId on that provider and leave Export__Nis2Soa__SigningPrivateKeyPem unset (delete the inline PEM). The SoA signer (Nis2SoaCryptoProviderSigner) resolves the key by id through the abstraction either way.
3. Set SigningAlgorithmIdto the algorithm your provider exposes (ES256 by default; the EU offline verifier supports ECDSA/RSA trust roots, not Ed25519 — keep an EC curve for the SoA surface). Set SigningProviderHint to the production provider id (e.g. eu.eidas.soft) instead of default.
4. Publish the real trust root. Replace the dev-stable TrustRoots__0 block with the production key’s public-key PEM + sha256 Fingerprint + validity window + the signer profile (tenant-regulatory-export). Keep TrustRootMode: "AirGap" so the bundle carries the trust root inline and verifies offline (§5).
5. Keep ExpectedSignerKeyIdaligned with SigningKeyId so a run that signs with the wrong key fails verification.
6. Authority read path (unchanged for production): AuthorityBaseUrl must point at the gateway, and Export__Nis2Soa__ServiceAccount__* supplies a client-credentials account (authority:tenants.read) so ExportCenter can mint its own token to read the tenant compliance profile when the run arrives over the messaging transport (the caller’s bearer is a signed identity envelope, not a forwardable header). Replace the lab’s seeded stellaops-cli-automation client/secret with a production service account.

4) Surface A verification key — capsule JWKS (offline verify)

The capsule-seal public key is published as JWKS so an air-gapped operator verifies a sealed capsule’s DSSE signature fully offline:

• Endpoint: GET /api/v1/evidence/capsules/signing-keys/jwks (anonymous — a public verification key is by definition distributable; no private material is exposed). Verified in CapsuleSigningKeyEndpoints.cs (JwksPath).
• How it stays correct across providers: the endpoint resolves the signer through the same SignatureProviderSelection.Resolve(CapsuleSeal, …) selection as signing, then calls ICryptoSigner.ExportPublicJsonWebKey(). So whichever sovereign provider the routine selects (dev bouncycastle.ed25519 in the lab; HSM / Vault / eu.eidas.soft / fips in production), the published x (and crv/alg/kid) always matches the key that signs. You do not republish anything when you swap providers — the endpoint re-derives the public JWK from the active provider.
• Honest degradation: if UseCryptoRegistry=false, KeyId is unset, or the provider key is not yet provisioned, the endpoint returns 503 (capsule_signing_key_unavailable) instead of a 500 — the offline-verify contract degrades honestly rather than publishing a wrong key.

Air-gap offline-verify procedure (operator):

1. Fetch the JWKS once from a connected boundary: GET …/signing-keys/jwks → save the keys[0] public JWK (kty/crv/x/kid/alg).
2. Export the sealed capsule: POST /api/v1/evidence/capsules/{id}/export → a zip with manifest.json, manifest.dsse.json, …
3. Offline: recompute the canonical content hash from manifest.json, then DSSE-verify the Ed25519 signature in manifest.dsse.json against the published x public key — no call back into the stack. Capsule DSSE signatures are reproducible across restarts for a fixed input + key, so this is a genuine air-gap deliverable.

5) Surface B verification key — SoA embedded trust root (offline verify)

The NIS2 SoA bundle carries its trust root inline (TrustRootMode: "AirGap"), so it verifies offline with no JWKS fetch:

• The signed run returns verification.isValid=true with verifiedKeyId, verifiedTrustRootId, and verifiedTrustRootFingerprint.

• The stella CLI verifies the bundle offline:

  dotnet src/Cli/StellaOps.Cli/bin/Debug/net10.0/StellaOps.Cli.dll \
    export nis2-soa --since <date> \
    --snapshot snapshot.json --tenant-profile tenant-profile.json \
    --signing-key-file <key> --signing-key-id nis2-soa-export-key \
    --output nis2-soa-bundle.json --overwrite --json
  # -> "offlineVerified": true
  

• In production, the TrustRoots__0__PublicKeyPem + Fingerprint you publish (§3.2) is the production provider’s public key — the bundle remains byte-identical across runs for a fixed input + key.

6) eIDAS QES (qualified) opt-in posture

eu.eidas.soft performs genuine ECDSA ES256/ES384 signing but is software-only, non-certified, QSCD not enforced. Under the eidas compliance profile the routine resolves ES256 for every purpose and the registry routes it to the first preferred provider — so the soft production on-ramp for the eu profile is three settings, no code change (see eidas-sim-crypto-demo.md §eIDAS production-signing on-ramp):

1. STELLAOPS_CRYPTO_COMPLIANCE_PROFILE=eidas (algorithm family → ES256/ES384).
2. EIDAS_SOFT_ALLOWED=1 and put eu.eidas.soft first in the registry preferred order.
3. Provision the signing KeyId on eu.eidas.soft (Vault-wrapped / PKCS#11) and point the signer’s KeyId/Provider at it — no raw private bytes in app config.

soft → qualified (QES). Making it a qualified eIDAS seal is a separate, deliberate step (intentionally not built speculatively). To upgrade — still no signer code change, because every signer only requests purpose → ES256 + a KeyId:

• Swap eu.eidas.soft for a QTSP/QSCD-backed provider (smart card / HSM) registered under the same registry preferred order.
• Flip Compliance.StrictValidation=true and Compliance.EnforceJurisdiction=true in etc/appsettings.crypto.eu.yaml (today both are off with a TODO: Replace with eidas.soft plugin when available).
• Supply qualified-trust evidence: QSCD attestation, QTSP qualified timestamps (RFC 3161 over a qualified TSA), TSL/LOTL trust list (offline eu-lotl.xml for air-gap), OCSP/CRL, and AdES (XAdES/PAdES/CAdES) packaging — per ETSI EN 319 412-1/2, ETSI TS 119 312, and the eIDAS QSCD/QTSP Bridge Contract.

Until a customer requires QES, the eu profile signs with the soft provider through the routine (production posture for the software lane) while the qualified posture stays an explicit, documented opt-in.

7) Before / after config diff (lab dev-stable → production sovereign)

7.1 EvidenceLocker capsule (evidence-locker-web + evidence-locker-worker)

  EvidenceLocker__Signing__Enabled: "true"
  EvidenceLocker__Signing__UseCryptoRegistry: "true"
  EvidenceLocker__Signing__KeyId: "evidence-locker-capsule-key"
  EvidenceLocker__Signing__AllowEphemeralKeyMaterial: "false"
-  # DEV-STABLE seed — lab/golden-path only:
-  EvidenceLocker__Signing__Algorithm: "ED25519"
-  EvidenceLocker__Signing__Provider: "bouncycastle.ed25519"
-  EvidenceLocker__Signing__KeyMaterial__Ed25519PrivateKeyBase64: "${STELLAOPS_EVIDENCELOCKER_CAPSULE_ED25519_SEED_B64:-4OtFcrPU+…NP3s=}"
+  # PRODUCTION: key pre-provisioned on the sovereign provider; no inline seed.
+  # Algorithm follows the active compliance profile (e.g. ES256 under eidas/fips);
+  # pin it only if you must. Provider = your sovereign provider id.
+  EvidenceLocker__Signing__Algorithm: "ES256"
+  EvidenceLocker__Signing__Provider: "eu.eidas.soft"   # or fips.ecdsa.soft / an HSM provider
+# (provision the KeyId on the provider; e.g. EIDAS_SOFT_ALLOWED=1 + registry preferred order)
+# STELLAOPS_CRYPTO_COMPLIANCE_PROFILE: "eidas"

Keep AllowEphemeralKeyMaterial=false. Removing the inline seed with the fallback off forces the key to come from the provider — the correct production failure mode. Under the default/world profile (capsule-seal → Ed25519) you would instead keep Algorithm: "ED25519" and provision an Ed25519 KeyId on the sovereign provider; the JWKS endpoint then publishes that provider’s Ed25519 public key unchanged.

7.2 ExportCenter NIS2 SoA (offlinekit-web, formerly export-web)

  Export__Nis2Soa__SigningKeyId: "nis2-soa-export-key"
  Export__Nis2Soa__SigningAlgorithmId: "ES256"
  Export__Nis2Soa__ExpectedSignerKeyId: "nis2-soa-export-key"
  Export__Nis2Soa__TrustRootMode: "AirGap"
-  Export__Nis2Soa__SigningProviderHint: "default"
-  # DEV-STABLE inline PEM — lab only; AddNis2SoaExportServices provisions it
-  # in-process when no ICryptoProviderRegistry is already registered:
-  Export__Nis2Soa__SigningPrivateKeyPem: |
-    -----BEGIN PRIVATE KEY-----
-    …
-    -----END PRIVATE KEY-----
-  Export__Nis2Soa__TrustRoots__0__TrustRootId: "stellaops-nis2-soa-dev-stable-2026"
-  Export__Nis2Soa__TrustRoots__0__PublicKeyPem: |
-    -----BEGIN PUBLIC KEY-----  # dev-stable public key
-    …
-    -----END PUBLIC KEY-----
-  Export__Nis2Soa__TrustRoots__0__Fingerprint: "sha256:8940…5459"
+  # PRODUCTION: register a sovereign-provider crypto registry (Vault / PKCS#11 /
+  # eu.eidas.soft) that OWNS nis2-soa-export-key; leave SigningPrivateKeyPem UNSET so
+  # AddNis2SoaExportServices skips inline provisioning (registry-present short-circuit).
+  Export__Nis2Soa__SigningProviderHint: "eu.eidas.soft"   # or fips.ecdsa.soft / HSM
+  Export__Nis2Soa__TrustRoots__0__TrustRootId: "<your-prod-trust-root-id>"
+  Export__Nis2Soa__TrustRoots__0__PublicKeyPem: |
+    -----BEGIN PUBLIC KEY-----  # the PRODUCTION provider public key
+    …
+    -----END PUBLIC KEY-----
+  Export__Nis2Soa__TrustRoots__0__Fingerprint: "sha256:<production-key-fingerprint>"
  Export__Nis2Soa__TrustRoots__0__NotBeforeUtc: "<prod validity start>"
  Export__Nis2Soa__TrustRoots__0__NotAfterUtc:  "<prod validity end>"
  Export__Nis2Soa__TrustRoots__0__SignerProfiles__0: "tenant-regulatory-export"
+  # Replace the seeded lab service account with a production one:
+  Export__Nis2Soa__ServiceAccount__ClientId: "<prod-export-service-account>"
+  Export__Nis2Soa__ServiceAccount__ClientSecret: "<prod-secret>"

8) Production preconditions checklist

Re-verify these per surface before approving a release that produces signed evidence. (Source of truth for each item: src/ paths cited above.)

• [ ] Profiles set for the region. STELLAOPS_CRYPTO_COMPLIANCE_PROFILE (algorithm family) + registry ActiveProfile/PreferredProviders; regional compose overlay (STELLAOPS_CRYPTO_PROFILE) if used. All three reconciled (§1.1).
• [ ] Chosen provider’s env gate enabled (EIDAS_SOFT_ALLOWED / FIPS_SOFT_ALLOWED / SM_SOFT_ALLOWED / GOST / PQ / HSM) and the provider appears first in the registry preferred order. sim.crypto.remote is not enabled.
• [ ] No cloud-managed KMS configured as a default — sovereign / Vault / HSM only.
• [ ] Capsule key evidence-locker-capsule-key provisioned on the provider; UseCryptoRegistry=true; inline Ed25519PrivateKeyBase64 removed; AllowEphemeralKeyMaterial=false; AllowUnsignedCapsules=false.
• [ ] SoA key nis2-soa-export-key owned by a pre-registered sovereign registry; SigningPrivateKeyPem unset; production TrustRoots__0 (PEM + fingerprint + validity + signer profile) published; ExpectedSignerKeyId aligned.
• [ ] JWKS GET /api/v1/evidence/capsules/signing-keys/jwks returns 200 and the keys[0].x matches the provider’s public key (not the lab seed-derived key).
• [ ] Round-trip both surfaces: seal a capsule and verify it fully offline against the JWKS key (no service call); run POST /v1/exports/nis2/soa/runs and confirm verification.isValid=true + offline CLI verify green.
• [ ] eIDAS posture explicit (soft vs qualified per §6); if QES is claimed, the StrictValidation/EnforceJurisdiction flags and QTSP/QSCD/TSL evidence are in place — otherwise the bundle is not presented as qualified.

9) Related docs

• Release checklist — Production signing
• Compliance signing: provider-selection routine & production handoff
• Compliance Golden-Path Demo — the residual “Production hardening” rows formalized here.
• eIDAS sim-crypto round-trip demo + production on-ramp
• Crypto profile configuration — region/profile selection + the stella crypto profiles select <profile> CLI surface.
• Cryptography module architecture — abstraction & purpose model.