Key Escrow and Recovery Runbook

Audience: security engineers and operators responsible for recovering critical cryptographic keys (root signing keys, HSM master keys, trust anchors) when primary access is lost.

This runbook documents the Shamir’s Secret Sharing key escrow and recovery primitives in Stella Ops: how a key is split among custodians, how a threshold of shares reconstructs it, and how recovery is gated by a dual-control ceremony.

Read the implementation-status note first. Key escrow currently ships as a library-level primitive — there is no CLI, HTTP API, or persisted configuration yet. The status block below is authoritative about what is and is not usable today; the CLI and REST examples later in this runbook are clearly marked as forward-looking sketches.

Sprint: SPRINT_20260112_018_CRYPTO_key_escrow_shamir

Implementation status (reconciled against src/ on 2026-05-31): Key escrow ships today as a library-level cryptographic primitive in StellaOps.Cryptography.KeyEscrow (project src/__Libraries/StellaOps.Cryptography, StellaOps.Cryptography.csproj). The following are implemented and unit/integration tested:

• ShamirSecretSharing — GF(2^8) split/combine/verify (KeyEscrow/ShamirSecretSharing.cs, KeyEscrow/GaloisField256.cs).
• KeyEscrowService : IKeyEscrowService — escrow/recover/status/list/revoke/re-escrow orchestration (KeyEscrow/KeyEscrowService.cs, KeyEscrow/IKeyEscrowService.cs).
• CeremonyAuthorizedRecoveryService — dual-control ceremony gating for recovery (KeyEscrow/CeremonyAuthorizedRecoveryService.cs).
• Contracts: IEscrowAgentStore, IKeyEscrowAuditLogger, and the KeyShare/KeyEscrowMetadata/KeyEscrowAuditEvent models (KeyEscrow/IEscrowAgentStore.cs, KeyEscrow/KeyEscrowModels.cs).

NOT yet implemented (do not rely on these — see the flagged sections below):

• No CLI. There is no stella escrow command (no escrow references exist under src/Cli).
• No HTTP/REST API. No escrow/recovery endpoints exist (the signer.example.com/api/v1/escrow examples below are illustrative only). The only related HTTP surface is the Signer dual-control ceremony endpoints (src/Attestor/StellaOps.Signer/StellaOps.Signer.WebService/Endpoints/CeremonyEndpoints.cs).
• No persistence / DI registration. IEscrowAgentStore and IKeyEscrowAuditLogger have no shipped concrete implementation or service registration; consumers must supply their own. The only implementations today are test doubles.
• No dedicated escrow auth scopes in StellaOps.Auth.Abstractions/StellaOpsScopes.cs.
• No escrow-config.yaml / custodians.yaml loader. The YAML in this runbook is a forward-looking proposal, not a parsed config surface.
• Share decryption during recovery is a TODO — KeyEscrowService.RecoverKeyAsync currently treats KeyShare.EncryptedData as already-decrypted share bytes (see “Key Recovery” caveat below).

Overview

Key escrow ensures critical cryptographic keys can be recovered if primary access is lost. Stella Ops uses Shamir’s Secret Sharing to split keys into shares distributed among trusted custodians (escrow agents).

Implemented capabilities (library level):

• M-of-N threshold recovery (any M shares reconstruct the key) via ShamirSecretSharing
• Share encryption at rest (AES-256-GCM; see caveat in “Share Security”)
• Custodian-based share distribution against an IEscrowAgentStore
• Integration with dual-control ceremonies (CeremonyAuthorizedRecoveryService)
• Audit event emission through IKeyEscrowAuditLogger

When to Use Key Escrow

Shamir Secret Sharing

How It Works

Shamir’s Secret Sharing splits a secret into N shares where any M shares can reconstruct the original:

Secret S → Split(S, M, N) → [Share₁, Share₂, ..., Shareₙ]

Any M shares → Combine → Secret S
Fewer than M shares → Cannot reconstruct

Configuration Parameters

ShamirSecretSharing.Split enforces 2 <= M <= N <= 255 (ValidateParameters in ShamirSecretSharing.cs). On the per-operation KeyEscrowOptions, Threshold (M) and TotalShares (N) are required(no effective default — the caller must supply them); ExpirationDays defaults to 365. The DefaultThreshold = 3 / DefaultTotalShares = 5 / DefaultExpirationDays = 365 values on KeyEscrowServiceOptions are declared but never read (see the “Options objects” caveat under Configuration); treat 3-of-5 as a recommended shape, not an automatic default.

Threshold Guidelines

Escrowing a Key

Escrow is invoked programmatically today via IKeyEscrowService. There is no shipped CLI or HTTP endpoint — see the flagged subsections below.

Via the library API (implemented)

// IKeyEscrowService is resolved by the host that wires up an IEscrowAgentStore + IKeyEscrowAuditLogger.
KeyEscrowResult result = await escrowService.EscrowKeyAsync(
    keyId: "root-signing-key-2026",
    keyMaterial: keyBytes,
    options: new KeyEscrowOptions
    {
        Threshold = 3,
        TotalShares = 5,
        ExpirationDays = 365,
        AgentIds = new[]
        {
            "custodian-1", "custodian-2", "custodian-3",
            "custodian-4", "custodian-5",
        },
        RequireDualControl = true,
    });

EscrowKeyAsync returns a KeyEscrowResult with the shape (see KeyEscrowModels.cs):

{
  "success": true,
  "keyId": "root-signing-key-2026",
  "shareIds": [ "<guid>", "<guid>", "<guid>", "<guid>", "<guid>" ],
  "threshold": 3,
  "totalShares": 5,
  "expiresAt": "2027-01-16T10:00:00Z",
  "error": null
}

There is no escrowIdconcept — escrow records are keyed by keyId (string), and individual shares carry an opaque ShareId (Guid). The esc-* / shr-* identifiers used in earlier drafts of this runbook do not exist.

Via CLI (NOT IMPLEMENTED — illustrative only)

No stella escrow command exists (src/Cli contains no escrow references). The example below is a forward-looking sketch of a possible CLI; do not run it.

# NOT IMPLEMENTED — roadmap sketch
stella escrow create \
  --key-id root-signing-key-2026 \
  --threshold 3 \
  --shares 5 \
  --custodians custodian-1,custodian-2,custodian-3,custodian-4,custodian-5 \
  --expires-in 365d \
  --reason "Annual key escrow for root signing key"

Via REST API (NOT IMPLEMENTED — illustrative only)

No escrow HTTP endpoint exists in Signer or any other service. The closest implemented surface is the Signer dual-control ceremony API (CeremonyEndpoints.cs), which can gate recovery but does not itself create escrows. The curl below is a forward-looking sketch.

# NOT IMPLEMENTED — roadmap sketch
curl -X POST https://signer.example.com/api/v1/escrow \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "keyId": "root-signing-key-2026",
    "threshold": 3,
    "totalShares": 5,
    "custodianIds": [
      "custodian-1", "custodian-2", "custodian-3",
      "custodian-4", "custodian-5"
    ],
    "expirationDays": 365,
    "reason": "Annual key escrow for root signing key"
  }'

Share Distribution

Distribution Methods

Custodian Requirements

An escrow agent (EscrowAgent in KeyEscrowModels.cs) carries AgentId, Name, Email, PublicKeyPem, IsActive, and RegisteredAt. Operationally, each custodian should:

1. Have a verified identity (Authority is the intended identity source; note there is no shipped Authority-to-escrow binding yet)
2. Complete escrow custodian training
3. Have secure share storage capability
4. Be geographically distributed (recommended)

Verifying Share Distribution (implemented via library)

Status is read with IKeyEscrowService.GetEscrowStatusAsync(keyId), which returns a KeyEscrowStatus (IKeyEscrowService.cs):

{
  "keyId": "root-signing-key-2026",
  "isEscrowed": true,
  "threshold": 3,
  "totalShares": 5,
  "validShares": 5,        // shares whose ExpiresAt is still in the future
  "createdAt": "2026-01-16T10:00:00Z",
  "expiresAt": "2027-01-16T10:00:00Z",
  "custodianIds": [ "custodian-1", "custodian-2", "custodian-3", "custodian-4", "custodian-5" ]
  // CanRecover (computed) = ValidShares >= Threshold
}

The stella escrow status --escrow-id ... CLI form is NOT IMPLEMENTED. Status is queried programmatically by keyId, not by an escrow ID.

Key Recovery

Prerequisites

Recovery requires:

1. Valid recovery request (incident, key loss, rotation) — KeyRecoveryRequest with KeyId, Reason, InitiatorId, AuthorizingCustodians, optional CeremonyId
2. Dual-control ceremony approval (when RequireDualControl is set on the escrow; enforced by requiring at least 2 AuthorizingCustodians)
3. Minimum M shares supplied to the service (shares.Count >= metadata.Threshold)
4. Secure recovery environment

Recovery Workflow (implemented via library)

The shipped flow is the ceremony-authorized path in CeremonyAuthorizedRecoveryService:

1. InitiateRecoveryAsync(request, initiator)
     - verifies the key is escrowed and not expired
     - creates a dual-control ceremony via ICeremonyAuthorizationProvider
     - returns a ceremonyId + RequiredApprovals + ExpiresAt
2. Ceremony reaches Approved state (out-of-band approvals)
3. ExecuteRecoveryAsync(ceremonyId, shares, executor)
     - verifies ceremony State == Approved and not expired
     - calls KeyEscrowService.RecoverKeyAsync with the collected shares
4. RecoverKeyAsync verifies each share's SHA-256 checksum, then
   reconstructs the secret via Lagrange interpolation (GaloisField256)
5. Ceremony marked Executed; KeyRecovered audit event logged

Important caveat — share decryption is a TODO. In the current KeyEscrowService.RecoverKeyAsync, the KeyShare.EncryptedData field is consumed as if it were already-decrypted share bytes (the code has // TODO: decrypt based on EncryptionInfo). The AES-256-GCM unwrap that pairs with EncryptShareAsync is not yet wired into recovery. Until that is closed, end-to-end recovery only round-trips in tests where shares are supplied in cleartext. Treat production recovery as not-yet-complete.

The lower-level IKeyEscrowService.RecoverKeyAsync can also be called directly (without a ceremony) for non-dual-control escrows.

Via CLI (NOT IMPLEMENTED — illustrative only)

No stella escrow recover ... command exists. Sketch only — do not run.

# NOT IMPLEMENTED — roadmap sketch
stella escrow recover init \
  --key-id root-signing-key-2026 \
  --reason "HSM failure - emergency key recovery" \
  --ceremony-required

stella escrow recover submit-share \
  --ceremony-id <ceremony-guid> \
  --share-file /secure/my-share.enc

stella escrow recover execute \
  --ceremony-id <ceremony-guid> \
  --output-key-file /secure/recovered-key.pem

Via REST API (NOT IMPLEMENTED — illustrative only)

No escrow/recovery HTTP endpoints exist. The Signer ceremony endpoints (CeremonyEndpoints.cs) are the only related surface and are not escrow-specific. Sketch only.

# NOT IMPLEMENTED — roadmap sketch
curl -X POST https://signer.example.com/api/v1/escrow/root-signing-key-2026/recover \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "reason": "HSM failure - emergency key recovery", "requireCeremony": true }'

Recovery Result (implemented)

RecoverKeyAsync returns a KeyRecoveryResult (KeyEscrowModels.cs):

{
  "success": true,
  "keyId": "root-signing-key-2026",
  "keyMaterial": "<byte[] — cleared after use>",
  "sharesUsed": 3,
  "error": null,
  "auditEventId": "<guid>"
}

There is no recoveryId, status, completedAt, or keyFingerprint field, and the key is not verified against a stored fingerprint by the service today.

Share Management

Custodian Share Storage

Custodians should store shares:

Share Format

The persisted share is the KeyShare record (KeyEscrowModels.cs). It is keyed by keyId + shareId (there is no escrowId):

{
  "shareId": "<guid>",
  "index": 1,                 // 1..N from Shamir splitting
  "encryptedData": "<byte[] — AES-256-GCM ciphertext || auth tag>",
  "keyId": "root-signing-key-2026",
  "threshold": 3,
  "totalShares": 5,
  "createdAt": "2026-01-16T10:00:00Z",
  "expiresAt": "2027-01-16T10:00:00Z",
  "custodianId": "custodian-1",
  "checksumHex": "<sha256 hex of the UNencrypted share data>",
  "schemaVersion": "1.0.0",
  "encryptionInfo": {
    "algorithm": "AES-256-GCM",
    "nonceBase64": "...",
    "authTagBase64": "...",
    "keyDerivationFunction": null,
    "saltBase64": null,
    "iterations": null
  }
}

Note: checksumHex is a lowercase hex SHA-256 digest (not the sha256:... prefixed form), computed over the unencrypted share bytes.

Share Rotation

Re-escrow keys periodically with IKeyEscrowService.ReEscrowKeyAsync(keyId, keyMaterial, options). This revokes (deletes) the existing shares for the key and creates a fresh set, then emits a KeyReEscrowed audit event. Passing options = null reuses the previous threshold / total-shares / dual-control settings (but not the previous ExpirationDays — re-escrow falls back to KeyEscrowOptions.ExpirationDays’s default of 365 unless you pass an explicit options).

Note: KeyEscrowMetadata carries a Generation field whose XML comment says it is “incremented on re-escrow”, but ReEscrowKeyAsync rebuilds metadata via EscrowKeyAsync and so the generation always resets to 1. Do not rely on Generation to count re-escrows today.

The stella escrow re-escrow ... CLI form is NOT IMPLEMENTED.

Audit Trail

Audit Events

Events are emitted as KeyEscrowAuditEvent records carrying a typed KeyEscrowAuditEventType enum value (see IEscrowAgentStore.cs). The enum members are:

The dotted string event names (escrow.created, recovery.completed, etc.) used in earlier drafts do not exist — events are the enum values above.

Each event also carries: EventId (Guid), KeyId, Timestamp, InitiatorId, optional Reason, optional CustodianIds, optional ShareCount, Success, optional Error, and optional CeremonyId.

Query Audit Logs

There is no stella audit query CLI escrow filter, and no shipped audit store/query API for escrow events. Persisting and querying KeyEscrowAuditEvents is the responsibility of whatever IKeyEscrowAuditLogger implementation the host supplies; no concrete implementation ships today.

Configuration

No YAML config loader is implemented. There is no escrow-config.yaml or custodians.yaml parser, and the library reads no config file. Behaviour is configured by the options objects below, supplied in code by the host. The YAML examples are forward-looking proposals (the originating sprint sketched a cryptography: escrow: block) and are not parsed today.

Options objects (code, not config files)

KeyEscrowServiceOptions (KeyEscrowService.cs):

Caveat — KeyEscrowServiceOptions is currently inert. The constructor stores the options, but KeyEscrowService never dereferences _options anywhere in the shipped code (verified 2026-05-31). Consequences:

• None of the four Default* / AutoDelete* properties take effect today. They are not consulted on any code path.
• The actual M / N / expiration used by EscrowKeyAsync come exclusively from the per-operation KeyEscrowOptions argument, whose Threshold and TotalShares are required (so a caller must always supply them) and whose ExpirationDays defaults to 365 on KeyEscrowOptions itself — not on KeyEscrowServiceOptions.
• AutoDeleteOnRecovery does not delete shares after recovery; to clear shares, call RevokeEscrowAsync (or ReEscrowKeyAsync) explicitly.

Treat the defaults in the table above as design intent, not current behaviour, until _options is wired into the service.

CeremonyAuthorizedRecoveryOptions (CeremonyAuthorizedRecoveryService.cs):

Per-operation overrides live on KeyEscrowOptions (Threshold, TotalShares, ExpirationDays, AgentIds, RequireDualControl, Metadata).

Proposed escrow settings (NOT IMPLEMENTED — roadmap)

# PROPOSED — not parsed by any code
cryptography:
  escrow:
    enabled: true
    defaultThreshold: 3
    defaultTotalShares: 5
    shareEncryptionKeyPath: "/etc/stellaops/escrow-encryption.key"
    shareRetentionDays: 365
    autoDeleteOnRecovery: false

Proposed custodian configuration (NOT IMPLEMENTED — roadmap)

# PROPOSED — not parsed by any code; agents are supplied via IEscrowAgentStore today
custodians:
  - id: custodian-1
    name: "Security Lead"
    email: security-lead@company.com
    publicKey: "-----BEGIN PUBLIC KEY-----..."

  - id: custodian-2
    name: "Key Officer A"
    email: key-officer-a@company.com
    publicKey: "-----BEGIN PUBLIC KEY-----..."

Security Considerations

Share Security

• Never transmit shares in plaintext.
• Encryption caveat: KeyEscrowService.EncryptShareAsync currently wraps each share with AES-256-GCM under a freshly generated random 256-bit key (GenerateKey()), and the code carries a comment that “in production, this would encrypt with the agent’s public key.” Per-custodian public-key wrapping (EscrowAgent.PublicKeyPem) is not yet implemented, and the random wrapping key is not currently persisted into ShareEncryptionInfo — which is part of why recovery treats the data as cleartext (see the Key Recovery caveat). Do not treat shares as recoverable via public-key unwrap today.
• Verify checksums (ChecksumHex, SHA-256 of the unencrypted share) before and after storage.
• Use secure channels for distribution.

Recovery Security

• Require dual-control ceremonies for critical keys (RequireDualControl / CeremonyAuthorizedRecoveryService).
• Limit the recovery time window (CeremonyExpirationMinutes, default 60).
• Verify the recovered key fingerprint out of band — the service does not currently compare the reconstructed key against a stored fingerprint.
• Audit all recovery attempts (KeyRecovered / RecoveryFailed events).

Custodian Security

• Verify custodian identity before share access
• Geographic distribution reduces collusion risk
• Rotate custodians periodically
• Train custodians on secure handling

Troubleshooting

Common Issues

Verification

ShamirSecretSharing.Verify(shares) attempts a reconstruction and returns true if a candidate set of shares is internally consistent (correct length, no duplicate/zero indices) — it does not reveal the secret. KeyEscrowService.RecoverKeyAsync independently re-checks each share’s SHA-256 ChecksumHex before combining.

The stella escrow verify-share / stella escrow test-recovery CLI commands are NOT IMPLEMENTED. Verification is performed programmatically via ShamirSecretSharing.Verify / the checksum check inside RecoverKeyAsync.

Emergency Procedures

Lost Share

If a custodian loses their share:

1. Verify at least M shares remain accessible
2. Re-escrow with new share set
3. Revoke compromised escrow
4. Document incident

Compromised Custodian

If a custodian is compromised:

1. Do NOT use their share for any recovery
2. Re-escrow immediately with new custodians
3. Revoke old escrow
4. Consider key rotation if threshold was exposed

Multiple Lost Shares

If fewer than M shares are available:

1. Key cannot be recovered via escrow
2. Use backup key if available
3. Generate new key and re-establish trust
4. Document as key loss incident

Related Documentation

• Incident Response Runbook
• Dual-Control Ceremony Runbook
• Key Rotation Runbook
• HSM Setup Runbook
• Cryptography Architecture