Dual-Control Ceremony Runbook

Audience: key custodians, security leads, and Signer/Attestor operators. Purpose: Run M-of-N threshold signing ceremonies for high-assurance key operations in Stella Ops, so that no single individual can perform a sensitive cryptographic action alone.

This runbook documents the implemented ceremony surface (HTTP API, authorization model, state machine, audit events, and configuration), and flags inline where earlier drafts described behaviour that is not yet shipped.

Sprint: SPRINT_20260112_018_SIGNER_dual_control_ceremonies

Module: Signer (consolidated under Attestor). Implementation lives in src/Attestor/StellaOps.Signer/StellaOps.Signer.Core/Ceremonies/ (orchestrator, state machine, models, audit events, options), src/Attestor/StellaOps.Signer/StellaOps.Signer.Infrastructure/Ceremonies/ (the HttpContextCeremonyApproverValidator, PostgresCeremonyRepository, and PostgresCeremonyAuditSink), and src/Attestor/StellaOps.Signer/StellaOps.Signer.WebService/Endpoints/CeremonyEndpoints.cs (HTTP surface; policy-to-scope wiring is in the WebService Program.cs). Persistence: signer.ceremonies / signer.ceremony_approvals / signer.ceremony_audit_log. The migration SQL (002_ceremony_runtime_state.sql) is sourced from the KeyManagement library at src/Attestor/__Libraries/StellaOps.Signer.KeyManagement/Migrations/.

Overview

Dual-control ceremonies ensure critical cryptographic operations require approval from multiple authorized individuals before execution. This prevents single points of compromise for sensitive operations. The implemented CeremonyOperationType enum covers:

• KeyGeneration — generate a new signing key
• KeyRotation — rotate an existing key
• KeyRevocation — revoke a key
• KeyExport — export a key (for escrow or backup)
• KeyImport — import a key from escrow or backup
• KeyRecovery — emergency key recovery

Trust-anchor updates are described elsewhere in this runbook as a use case but are not a ceremony operation type in code; only the six Key* operations above exist.

When Ceremonies Are Required

The implemented threshold model is a single global default plus optional per-operation overrides (Signer:Ceremonies config — see Configuration). There are no built-in operation-specific defaults baked into code beyond the global DefaultThreshold.

The “2-of-3 / 2-of-4 / 3-of-5” per-operation defaults in earlier drafts of this runbook were aspirational — they are not present in CeremonyOptions. Set per-operation thresholds explicitly in configuration. The threshold is M (ThresholdRequired); there is no separate “required approvers” / N value tracked in code — N is governed by who is authorized to approve.

Ceremony Lifecycle

State Machine

         +------------------+
         |     Pending      |
         +--------+---------+
                  |
                  | Approvals collected
                  v
    +-------------+-------------+
    |   PartiallyApproved      |
    +-------------+-------------+
                  |
                  | Threshold reached
                  v
         +--------+---------+
         |     Approved     |
         +--------+---------+
                  |
                  | Execute
                  v
         +--------+---------+
         |     Executed     |
         +------------------+

   Alternative paths (all from CeremonyStateMachine.IsValidTransition):
   - Pending -> Approved             (direct, when threshold = 1)
   - Pending -> Expired              (timeout)
   - Pending -> Cancelled            (cancel)
   - PartiallyApproved -> PartiallyApproved  (further approvals, threshold not yet reached)
   - PartiallyApproved -> Expired
   - PartiallyApproved -> Cancelled
   - Approved -> Expired             (execution window expired)
   - Approved -> Cancelled
   Terminal states (Executed / Expired / Cancelled) have no outgoing transitions.

State Descriptions

Creating a Ceremony

CLI not implemented. There is currently no stella ceremony CLI command group. The only supported surface is the HTTP API below. CLI examples in earlier drafts of this runbook are roadmap-only and do not correspond to any code under src/Cli.

Via API

POST /api/v1/ceremonies. Requires the signer:sign scope (registered as the ceremony:create policy). The request body maps to CreateCeremonyRequestDto:

• operationType (required, string): one of KeyGeneration, KeyRotation, KeyRevocation, KeyExport, KeyImport, KeyRecovery. Snake_case (key_rotation) is also accepted; the hyphenated form (key-rotation) is not.
• thresholdRequired (required, int): the M in M-of-N. Overrides the configured default.
• timeoutMinutes (optional, int): expiry window in minutes; falls back to the configured default (60) when omitted. There is no expiresAt absolute-timestamp input.
• payload (optional object): keyId, algorithm, keySize, keyUsages[], reason, metadata{}.
• description (optional string): human-readable label (this is the field; there is no subject).
• tenantId (optional string): for multi-tenant deployments.

curl -X POST https://signer.example.com/api/v1/ceremonies \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "operationType": "KeyRotation",
    "thresholdRequired": 2,
    "timeoutMinutes": 1440,
    "description": "Root signing key Q1-2026",
    "payload": {
      "keyId": "root-2026-q1",
      "algorithm": "ecdsa-p384"
    }
  }'

Response

201 Created with Location: /api/v1/ceremonies/{ceremonyId} and a CeremonyResponseDto body. ceremonyId is a GUID. Field names match the DTO exactly:

{
  "ceremonyId": "3f2504e0-4f89-41d3-9a0c-0305e82c3301",
  "operationType": "KeyRotation",
  "state": "Pending",
  "thresholdRequired": 2,
  "thresholdReached": 0,
  "initiatedBy": "admin@company.com",
  "initiatedAt": "2026-01-16T10:00:00Z",
  "expiresAt": "2026-01-17T10:00:00Z",
  "executedAt": null,
  "description": "Root signing key Q1-2026",
  "tenantId": null,
  "payload": {
    "keyId": "root-2026-q1",
    "algorithm": "ecdsa-p384",
    "keySize": null,
    "keyUsages": null,
    "reason": null,
    "metadata": null
  },
  "approvals": []
}

Approving a Ceremony

Prerequisites

The ceremony:approve policy requires the signer:sign scope. On top of the scope check, the HttpContextCeremonyApproverValidator enforces, in order:

1. A non-empty approver identity and a non-empty approval signature (an empty signature is rejected with InvalidSignature).
2. An authenticated principal whose sub claim matches the supplied approver identity.
3. A non-empty tenant claim.
4. The signer audience in the token’s aud claim(s).
5. At least one of the signer:sign or signer:admin scopes.
6. The caller must not have already approved this ceremony (enforced separately by the orchestrator via the uq_ceremony_approvals_identity unique constraint → DuplicateApproval).

The validator checks scope/identity/tenant/audience — it does not consult an external “allowed approvers list” or approvers.yaml. The RequiredRoles per-operation option exists in CeremonyOptions but is not consulted by the implemented validator. The signature bytes are stored and required to be present, but cryptographic verification of the signature against a key is not performed by this validator (it validates the caller’s authenticated context, not the signature math). See Decisions & gaps.

Via API

CLI approval (stella ceremony approve --sign) is not implemented.

POST /api/v1/ceremonies/{ceremonyId}/approve. The {ceremonyId} path segment is a GUID. Body maps to ApproveCeremonyRequestDto:

• signature (string, base64): required; rejected with 400 if missing or not valid base64.
• reason (optional string): stored as the approval reason.
• signingKeyId (optional string): key ID used to sign the approval.

curl -X POST https://signer.example.com/api/v1/ceremonies/3f2504e0-4f89-41d3-9a0c-0305e82c3301/approve \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "reason": "Reviewed rotation plan, verified key parameters",
    "signature": "base64-encoded-signature",
    "signingKeyId": "user-signing-key-456"
  }'

Approval Response

200 OK returns the full updated CeremonyResponseDto(not a standalone approval object). The new approval appears in the approvals[] array; there is no signatureValid field and no currentApprovals/threshold shorthand (use thresholdReached/thresholdRequired):

{
  "ceremonyId": "3f2504e0-4f89-41d3-9a0c-0305e82c3301",
  "operationType": "KeyRotation",
  "state": "PartiallyApproved",
  "thresholdRequired": 2,
  "thresholdReached": 1,
  "initiatedBy": "admin@company.com",
  "initiatedAt": "2026-01-16T10:00:00Z",
  "expiresAt": "2026-01-17T10:00:00Z",
  "executedAt": null,
  "description": "Root signing key Q1-2026",
  "tenantId": null,
  "payload": { "keyId": "root-2026-q1", "algorithm": "ecdsa-p384" },
  "approvals": [
    {
      "approvalId": "a1b2c3d4-0000-0000-0000-000000000001",
      "approverIdentity": "security-lead@company.com",
      "approvedAt": "2026-01-16T11:30:00Z",
      "reason": "Reviewed rotation plan, verified key parameters"
    }
  ]
}

Failure status codes: 404 (not found), 409 (DuplicateApproval, AlreadyExecuted, Cancelled, Expired), 403 (UnauthorizedApprover), 400 (InvalidSignature and other).

Executing a Ceremony

Once the approval threshold is reached the ceremony transitions to Approved and can be executed. Execution requires the signer:admin scope (the ceremony:execute policy maps to signer:admin, not to the initiator). Execution is rejected with 409 if the ceremony is not in the Approved state or its execution window has expired.

CLI execution (stella ceremony execute) is not implemented.

Via API

POST /api/v1/ceremonies/{ceremonyId}/execute (GUID path segment):

curl -X POST https://signer.example.com/api/v1/ceremonies/3f2504e0-4f89-41d3-9a0c-0305e82c3301/execute \
  -H "Authorization: Bearer $TOKEN"

Execution Response

Important — execution does not perform the key operation today. The implemented CeremonyOrchestrator.ExecuteCeremonyAsync marks the ceremony Executed, sets executedAt, emits a signer.ceremony.executed audit event, and returns the updated ceremony record. It does not generate/rotate/revoke a key or return any key material. The result block with publicKey/fingerprint/activatedAt shown in earlier drafts is not produced — wiring the approved ceremony to the actual key operation is an open gap (see Decisions & gaps).

200 OK returns the full CeremonyResponseDto with state: "Executed" and a populated executedAt:

{
  "ceremonyId": "3f2504e0-4f89-41d3-9a0c-0305e82c3301",
  "operationType": "KeyRotation",
  "state": "Executed",
  "thresholdRequired": 2,
  "thresholdReached": 2,
  "initiatedBy": "admin@company.com",
  "initiatedAt": "2026-01-16T10:00:00Z",
  "expiresAt": "2026-01-17T10:00:00Z",
  "executedAt": "2026-01-16T14:00:00Z",
  "description": "Root signing key Q1-2026",
  "tenantId": null,
  "payload": { "keyId": "root-2026-q1", "algorithm": "ecdsa-p384" },
  "approvals": [ /* ... */ ]
}

Monitoring Ceremonies

CLI monitoring (stella ceremony list / stella ceremony status) is not implemented. Use the HTTP API. Listing requires the group-level ceremony:read policy (signer:read scope).

List Ceremonies

GET /api/v1/ceremonies supports query filters state, operationType, initiatedBy, tenantId, limit (default 50), and offset (default 0). state accepts a single value (pending, partiallyapproved/partially_approved, approved, executed, expired, cancelled); an unrecognized value is silently ignored (treated as no state filter). The response is CeremonyListResponseDto (ceremonies[], totalCount, limit, offset).

curl "https://signer.example.com/api/v1/ceremonies?state=pending&limit=50" \
  -H "Authorization: Bearer $TOKEN"

The comma-separated multi-state filter (?state=pending,partially-approved) shown in earlier drafts is not supported — only one state value is parsed.

Check Ceremony Status

GET /api/v1/ceremonies/{ceremonyId} (GUID) returns the CeremonyResponseDto, or 404 if not found.

curl "https://signer.example.com/api/v1/ceremonies/3f2504e0-4f89-41d3-9a0c-0305e82c3301" \
  -H "Authorization: Bearer $TOKEN"

Cancelling a Ceremony

Ceremonies can be cancelled while in Pending, PartiallyApproved, or Approved state. Cancellation requires the signer:admin scope (the ceremony:cancel policy maps to signer:admin). The cancellation reason is passed as a query-string parameter, not a request body. CLI cancellation (stella ceremony cancel) is not implemented.

curl -X DELETE "https://signer.example.com/api/v1/ceremonies/3f2504e0-4f89-41d3-9a0c-0305e82c3301?reason=Postponed%20due%20to%20schedule%20conflict" \
  -H "Authorization: Bearer $TOKEN"

Returns 204 No Content on success. signer:admin callers may cancel; there is no initiator-only fast path in the implemented authorization (the policy is purely scope-based). 404 is returned for an unknown ceremony. Note that cancelling a ceremony already in a terminal state (Executed/Expired/Cancelled) returns 400, not 409: CancelCeremonyAsync yields CeremonyErrorCode.InternalError for any non-cancellable state, and the endpoint’s status map falls through to 400 for that code. (The route declares a 409 response shape, but the implemented cancel path does not currently produce it.)

Audit Events

All ceremony actions are written to signer.ceremony_audit_log (and emitted through the ICeremonyAuditSink). The event-type constants in CeremonyAuditEvents are:

The HTTP endpoints are additionally Audited via the platform audit pipeline with AuditModules.Attestor and actions create_ceremony, approve_ceremony, execute_ceremony, cancel_ceremony (AuditActions.Attestor).

Audit Event Structure

Fields follow the CeremonyAuditEvent record hierarchy. The base record carries eventType, ceremonyId (GUID), operationType, timestamp, actor, and optional tenantId/traceId. The approval event (CeremonyApprovedEvent) adds approver, approvalCount, thresholdRequired, approvalReason, and thresholdReached (bool). Note the field is operationType (not ceremonyType), and counts are approvalCount/thresholdRequired (there is no threshold, currentApprovals, approvalId, signatureAlgorithm, or signatureKeyId on the audit event):

{
  "eventType": "signer.ceremony.approved",
  "ceremonyId": "3f2504e0-4f89-41d3-9a0c-0305e82c3301",
  "operationType": "KeyRotation",
  "timestamp": "2026-01-16T11:30:00Z",
  "actor": "security-lead@company.com",
  "tenantId": null,
  "traceId": null,
  "approver": "security-lead@company.com",
  "approvalCount": 1,
  "thresholdRequired": 2,
  "approvalReason": "Reviewed rotation plan, verified key parameters",
  "thresholdReached": false
}

Query CLI not implemented. There is no stella audit query command. Query the signer.ceremony_audit_log table directly or via the platform audit/Timeline surface.

Configuration

Ceremony Settings

Ceremony configuration binds to the CeremonyOptions class under the Signer:Ceremonies configuration section. The implemented keys are:

The <type> keys are the lowercased enum names: keygeneration, keyrotation, keyrevocation, keyexport, keyimport, keyrecovery.

# appsettings (Signer service) — Signer:Ceremonies
Signer:
  Ceremonies:
    Enabled: true
    DefaultThreshold: 2
    ExpirationMinutes: 60
    Operations:
      keyrotation:
        Threshold: 2
        ExpirationMinutes: 1440
      keyrevocation:
        Threshold: 3

Not implemented. The earlier defaultTimeout/maxTimeout/requireSignedApprovals keys and the minimum/maximum threshold bounds do not exist in CeremonyOptions. There is a single DefaultThreshold plus per-operation Threshold overrides (each range-validated 1–10); there is no enforced minimum/maximum band per operation. RequiredRoles is defined on the options object and exposed via GetRequiredRoles, but the implemented HttpContextCeremonyApproverValidator does not consult it.

Approver Configuration

Not implemented. There is no approvers.yaml / approverGroups mechanism in code. Authorization to approve is governed entirely by token scope (signer:sign/signer:admin), audience (signer), tenant claim, and a sub-claim match against the approver identity (see Approving a Ceremony → Prerequisites). The block below is roadmap-only.

# ROADMAP ONLY — not consumed by any implemented code path
approverGroups:
  - name: key-custodians
    members:
      - security-lead@company.com
      - ciso@company.com
    operations:
      - keyrotation
      - keyrevocation

Notifications

Options-only — no dispatcher wired. A CeremonyNotificationConfig exists on CeremonyOptions (Signer:Ceremonies:Notifications) with the flags below, but there is no implemented code path that consumes these flags to send notifications, and there is no notifier-config.yaml integration for ceremonies. Treat ceremony notifications as roadmap. Audit events (signer.ceremony.*) are the implemented signal — alert off the audit log / audit pipeline.

The defined (but not yet dispatched) notification option fields are:

The “75% and 90% of timeout” two-stage warning in earlier drafts is not implemented; only a single ExpirationWarningMinutes field exists (and, as noted, no dispatcher reads it).

# Signer:Ceremonies:Notifications (options shape; not yet dispatched)
Signer:
  Ceremonies:
    Notifications:
      Channels: [ "email" ]
      NotifyOnCreate: true
      NotifyOnApproval: true
      NotifyOnExpirationWarning: true
      ExpirationWarningMinutes: 15

Security Best Practices

Approver Requirements

• Provision enough scope-bearing identities to satisfy the configured threshold M without relying on any single individual. Approver eligibility today is governed by token scope (signer:sign / signer:admin) and tenant — not by a named approver list (see Decisions & gaps).
• Distribute approver identities across security domains.
• Require hardware-backed credentials for the tokens used to approve.
• Rotate approver credentials/keys on your normal cadence.

Ceremony Hygiene

• Use a descriptive description for audit clarity.
• Set reasonable timeoutMinutes (the option is range-validated 5–1440 minutes).
• Document approval reasons thoroughly.
• Review executed ceremonies regularly via signer.ceremony_audit_log.

Monitoring

There is no built-in ceremony alerting engine; build alerts off the emitted audit events (signer.ceremony.*) in your monitoring stack. Useful signals derived from the implemented events:

The alerts: YAML schema in earlier drafts is illustrative — it is not a recognized Signer configuration block.

Troubleshooting

Common Issues

Signature / Authorization Failures

The stella auth keys / stella sign / stella verify CLI commands referenced in earlier drafts are not part of the implemented ceremony surface. To diagnose approval rejections, inspect the signer.ceremony.approval_rejected audit event (it records errorCode and rejectionReason) and verify the approver token carries: the signer audience, a tenant claim, a sub matching the submitted approver identity, and the signer:sign or signer:admin scope.

Emergency Procedures

Stuck Ceremony

If a ceremony is stuck (approvers unavailable):

1. Cancel the stuck ceremony
2. Create new ceremony with available approvers
3. Document the situation in audit notes

Compromised Approver

If an approver’s credentials are compromised:

1. Revoke the compromised token/credential (Authority) so its signer:* scopes no longer issue.
2. Cancel any pending or partially-approved ceremonies the identity touched (DELETE with a reason).
3. Review recent signer.ceremony.approved audit events for anomalies attributable to the identity.
4. Rotate the affected signing key(s) as appropriate.
5. Document in the incident report.

Step 4 references “approver groups” in earlier drafts — there is no approver-group store to remove from. Access is removed by revoking the identity’s scope grant in Authority.

Decisions & gaps

The following are deliberate accuracy notes about the implemented ceremony surface versus earlier aspirational documentation:

• Authorization is scope-based, not initiator/role-based. ceremony:read→signer:read, ceremony:create/ceremony:approve→signer:sign, ceremony:execute/ceremony:cancel→ signer:admin. These are ASP.NET policy names; the scope claim values are the signer:* catalog entries (StellaOpsScopes.cs). There is no ceremony:* scope in the scope catalog.
• Execution does not perform the key operation. ExecuteCeremonyAsync only transitions state to Executed, stamps executedAt, and emits an audit event. Binding an approved ceremony to the actual generate/rotate/revoke/export/import/recover operation (and returning key material) is an open implementation gap.
• Approver validation checks context, not signature math. The signature bytes are required and stored, but HttpContextCeremonyApproverValidator validates the authenticated caller (sub, tenant, audience, scope) rather than cryptographically verifying the supplied signature against a key.
• No CLI surface. No stella ceremony …, stella audit query, stella auth keys, or stella sign commands exist for ceremonies. HTTP is the only implemented interface.
• No notification dispatcher. CeremonyNotificationConfig options exist but nothing consumes them; there is no notifier-config.yaml ceremony integration.
• No approver-group / approvers.yaml mechanism, and Operations[*].RequiredRoles is defined but not enforced by the implemented validator.

Related Documentation

• Key Rotation Runbook
• HSM Setup Runbook
• Signer Architecture (Signer is consolidated under Attestor)
• Break-Glass Runbook