Promotion State Machine

Overview

Promotions move releases through environments (Dev -> Staging -> Production). The promotion state machine manages the lifecycle from request to completion.

For policy-driven evidence gate inputs, checks, and outcomes, see evidence-based-release-gates.md.

Evidence-gate outcomes include explicit hold_async and escalate semantics in decision records:

• hold_async maps to DecisionOutcome.HoldAsync for asynchronous evidence completion.
• escalate maps to DecisionOutcome.Escalate when retry budgets are exhausted or lane policy requires human disposition.

Emergency Hotfix Freeze Exceptions

Freeze-window exemptions are evidence-bearing release controls, not boolean overrides. When allowExemptions=true, the freeze gate may pass during an active freeze only when the exemption evidence includes a hotfix exception id, owner, reason, ticket, scope, expiry, and approver. The gate returns these fields in the gate details as hotfixExceptionId, exceptionOwnerId, exceptionReason, exceptionTicket, exceptionScope, exceptionExpiresAt, and exceptionApprovedBy.

Legacy exemption providers that can only answer “yes/no” are treated as incomplete evidence. They fail closed with exceptionEvidenceComplete=false and exceptionUnavailableReason=legacy_exemption_without_structured_evidence instead of allowing an emergency hotfix promotion without audit context.

The environment freeze-window exemption API follows the same rule. Exemption grants must persist the release bundle id, reason, ticket, scope, owner, granting actor, approving actor, approval time, expiry, and optional incident link. IsEnvironmentFrozenAsync(environmentId, releaseBundleId) only treats an exemption as a bypass when those required fields are present and the expiry is still in the future. Older rows or adapters that only prove a boolean exemption remain frozen until they are replaced by a fully evidenced exception.

The console hotfix detail surface must show the same exception evidence fields alongside the gate outcomes: exception id, ticket, scope, owner, approver, expiry, reason, evidence thread, release run id, and patched artifact digest. The exception is displayed as evidence for the freeze gate; it does not hide security, SBOM, reachability, policy, or approval gates.

Promotion States

┌─────────────────────────────────────────────────────────────────────────────┐
│                    PROMOTION STATE MACHINE                                  │
│                                                                             │
│                         ┌──────────────────┐                               │
│                         │ PENDING_APPROVAL │ (initial)                     │
│                         └────────┬─────────┘                               │
│                                  │                                          │
│               ┌──────────────────┼──────────────────┐                      │
│               │                  │                  │                      │
│               ▼                  ▼                  ▼                      │
│      ┌────────────────┐ ┌────────────────┐ ┌────────────────┐             │
│      │   REJECTED     │ │  PENDING_GATE  │ │   CANCELLED    │             │
│      └────────────────┘ └────────┬───────┘ └────────────────┘             │
│                                  │                                          │
│                                  │ gates pass                               │
│                                  ▼                                          │
│                         ┌────────────────┐                                 │
│                         │    APPROVED    │                                 │
│                         └────────┬───────┘                                 │
│                                  │                                          │
│                                  │ start deployment                         │
│                                  ▼                                          │
│                         ┌────────────────┐                                 │
│                         │   DEPLOYING    │                                 │
│                         └────────┬───────┘                                 │
│                                  │                                          │
│               ┌──────────────────┼──────────────────┐                      │
│               │                  │                  │                      │
│               ▼                  ▼                  ▼                      │
│      ┌────────────────┐ ┌────────────────┐ ┌────────────────┐             │
│      │    FAILED      │ │   DEPLOYED     │ │  ROLLED_BACK   │             │
│      └────────────────┘ └────────────────┘ └────────────────┘             │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

State Definitions

Approved → Deploying: the durable dispatch (as built)

Status: Implemented (Sprint 20260703_002 / DTC-8). The TypeScript sections below this one are illustrative design pseudocode; the as-built transition is described here and verified by src/ReleaseOrchestrator/__Tests/StellaOps.ReleaseOrchestrator.Integration.Tests/DeploymentDispatchDurableTransitionTests.cs and, through the production deployment engine, by DeploymentDispatchInterruptionTests.cs in the same project.

The approved → deploying edge is durable, not event-driven-best-effort:

1. Atomic intent persistence. Every surface that flips an approval row to approved (manual quorum endpoints, batch approve, dashboard/v2 decision endpoints, and the PromotionRequestProcessor auto-approve path) writes through DispatchOnApprovalReleaseTruthRepository, which persists a deployment-dispatch intent into release_orchestrator.deployment_dispatches in the same Postgres transaction as the approval upsert (PostgresDeploymentDispatchRepository.UpsertApprovalWithDispatchIntentAsync). The intent is unique per approval, so double-approves enqueue exactly once.
2. Durable dispatcher. The hosted DeploymentDispatchProcessor polls due intents (claim lease + FOR UPDATE SKIP LOCKED) and starts the deployment through ReleaseDeploymentInitiator — the same validation + engine entry POST /api/v1/release-orchestrator/releases/{id}/deploy uses, so the fail-closed ReleaseTruthGuard checks (approved quorum, persisted passing gate decision, verified evidence) and the deploy-engine path are identical regardless of which surface initiates the deployment.
3. Bounded retry, visible failure. Failures retry with deterministic exponential backoff up to MaxAttempts. Terminal failure marks the dispatch row failed, sets the release status to dispatch_failed, and appends a deployment_dispatch_failed release event — the transition is never silently dropped. (Historically the only trigger was a fire-and-forget HTTP call to the workflow engine that swallowed failures; that call remains as a workflow-engine notification only.)
4. Interruption, duplicate delivery and response loss. A claimed intent stays invisible to other instances for the claim lease, so dispatchers that poll at the same moment start the deployment once. Before any approval or release check, the dispatcher looks for a deployment this intent already started (same release, started at or after the intent, and recorded for the same approval; a record without an approval matches by environment, and a record of another approval never matches) and records the dispatch against it. Its deployment_dispatched release event then says that the dispatch was linked to an already-started deployment and names who started it (the dispatcher in an earlier attempt, or an operator’s deploy request for the same approval); only a deployment the claim itself started reads as auto-dispatched. A service stop between the engine start and the dispatch record therefore resolves on the first claim after the lease: no second start, and the dispatch is not recorded as cancelled although the engine has already moved the approval to deploying or deployed. When the dispatch record or its release event cannot be written after the engine started the deployment, or while a claim links an already-started deployment, the attempt is not a dispatch failure: the record is written again once, and otherwise the lease leaves the row for the next claim to link; dispatch_failed and deployment.failed are not produced once the engine has returned the started deployment or a claim has found it.
5. Kill-switch. ReleaseOrchestrator:AutoDeployOnApproval (default ON) disables both the enqueue and the dispatcher; see Configuration Reference.

State Transitions

Valid Transitions

const validTransitions: Record<PromotionStatus, PromotionStatus[]> = {
  pending_approval: ["pending_gate", "approved", "rejected", "cancelled"],
  pending_gate: ["approved", "rejected", "cancelled"],
  approved: ["deploying", "cancelled"],
  deploying: ["deployed", "failed", "rolled_back"],
  rejected: [],  // terminal
  cancelled: [], // terminal
  deployed: [],  // terminal (for this promotion)
  failed: ["rolled_back"],  // can trigger rollback
  rolled_back: [] // terminal
};

Transition Events

interface PromotionTransition {
  promotionId: UUID;
  fromState: PromotionStatus;
  toState: PromotionStatus;
  trigger: TransitionTrigger;
  triggeredBy: UUID;  // user or system
  timestamp: DateTime;
  details: object;
}

type TransitionTrigger =
  | "approval_granted"
  | "approval_rejected"
  | "gate_passed"
  | "gate_failed"
  | "deployment_started"
  | "deployment_completed"
  | "deployment_failed"
  | "rollback_triggered"
  | "rollback_completed"
  | "user_cancelled";

Promotion Flow

1. Request Promotion

async function requestPromotion(request: PromotionRequest): Promise<Promotion> {
  // Validate release exists and is ready
  const release = await getRelease(request.releaseId);
  if (release.status !== "ready" && release.status !== "deployed") {
    throw new Error("Release not ready for promotion");
  }

  // Validate target environment
  const environment = await getEnvironment(request.targetEnvironmentId);

  // Check freeze windows
  if (await isEnvironmentFrozen(environment.id)) {
    throw new Error("Environment is frozen");
  }

  // Determine initial state
  const requiresApproval = environment.requiredApprovals > 0;
  const initialStatus = requiresApproval ? "pending_approval" : "pending_gate";

  // Create promotion
  const promotion = await createPromotion({
    releaseId: request.releaseId,
    sourceEnvironmentId: release.currentEnvironmentId,
    targetEnvironmentId: environment.id,
    status: initialStatus,
    requestedBy: request.userId,
    requestReason: request.reason
  });

  // Emit event
  await emitEvent("promotion.requested", promotion);

  return promotion;
}

2. Approval Phase

async function processApproval(
  promotionId: UUID,
  approverId: UUID,
  action: "approve" | "reject",
  comment?: string
): Promise<Promotion> {
  const promotion = await getPromotion(promotionId);
  const environment = await getEnvironment(promotion.targetEnvironmentId);

  // Validate approver can approve
  await validateApproverPermission(approverId, environment.id);

  // Check separation of duties
  if (environment.requireSeparationOfDuties) {
    if (approverId === promotion.requestedBy) {
      throw new Error("Separation of duties violation: requester cannot approve");
    }
  }

  // Record approval
  await recordApproval({
    promotionId,
    approverId,
    action,
    comment
  });

  if (action === "reject") {
    return await transitionState(promotion, "rejected", {
      trigger: "approval_rejected",
      triggeredBy: approverId,
      details: { reason: comment }
    });
  }

  // Check if all required approvals received
  const approvalCount = await countApprovals(promotionId);
  if (approvalCount >= environment.requiredApprovals) {
    return await transitionState(promotion, "pending_gate", {
      trigger: "approval_granted",
      triggeredBy: approverId
    });
  }

  return promotion;
}

3. Gate Evaluation

async function evaluateGates(promotionId: UUID): Promise<GateEvaluationResult> {
  const promotion = await getPromotion(promotionId);
  const environment = await getEnvironment(promotion.targetEnvironmentId);
  const release = await getRelease(promotion.releaseId);

  // The live WebApi path resolves approval-policy, sbom-readiness,
  // attestation-gate, reachability-gate, and installed custom providers
  // through IGateProviderRegistry.
  const gateResults = await pluginBackedGateDecisionEvaluator.evaluate(
    release,
    environment,
    promotion
  );

  // Aggregate results
  const allPassed = gateResults.every(g => g.passed);
  const blockingFailures = gateResults.filter(g => !g.passed && g.blocking);

  // Create decision record
  const decisionRecord = await createDecisionRecord({
    promotionId,
    gateResults,
    decision: allPassed ? "allow" : "block",
    decidedAt: new Date()
  });

  // Transition state
  if (allPassed) {
    await transitionState(promotion, "approved", {
      trigger: "gate_passed",
      triggeredBy: "system",
      details: { decisionRecordId: decisionRecord.id }
    });
  } else {
    await transitionState(promotion, "rejected", {
      trigger: "gate_failed",
      triggeredBy: "system",
      details: { blockingGates: blockingFailures }
    });
  }

  return { passed: allPassed, gateResults, decisionRecord };
}

4. Deployment Execution

async function executeDeployment(promotionId: UUID): Promise<DeploymentJob> {
  const promotion = await getPromotion(promotionId);

  // Transition to deploying
  await transitionState(promotion, "deploying", {
    trigger: "deployment_started",
    triggeredBy: "system"
  });

  // Generate artifacts
  const artifacts = await generateArtifacts(promotion);

  // Create deployment job
  const job = await createDeploymentJob({
    promotionId,
    releaseId: promotion.releaseId,
    environmentId: promotion.targetEnvironmentId,
    artifacts
  });

  // Execute via workflow or direct
  const workflowRun = await startDeploymentWorkflow(job);

  // Update promotion with workflow reference
  await updatePromotion(promotionId, { workflowRunId: workflowRun.id });

  return job;
}

5. Completion Handling

async function handleDeploymentCompletion(
  jobId: UUID,
  status: "succeeded" | "failed"
): Promise<Promotion> {
  const job = await getDeploymentJob(jobId);
  const promotion = await getPromotion(job.promotionId);

  if (status === "succeeded") {
    // Generate evidence packet
    const evidence = await generateEvidencePacket(promotion, job);

    // Update release environment state
    await updateReleaseEnvironmentState({
      releaseId: promotion.releaseId,
      environmentId: promotion.targetEnvironmentId,
      status: "deployed",
      promotionId: promotion.id,
      evidenceRef: evidence.id
    });

    return await transitionState(promotion, "deployed", {
      trigger: "deployment_completed",
      triggeredBy: "system",
      details: { evidencePacketId: evidence.id }
    });
  } else {
    return await transitionState(promotion, "failed", {
      trigger: "deployment_failed",
      triggeredBy: "system",
      details: { jobId, error: job.errorMessage }
    });
  }
}

Decision Record

Every promotion produces a decision record:

interface DecisionRecord {
  id: UUID;
  promotionId: UUID;
  decision: "allow" | "block";
  decidedAt: DateTime;

  // Inputs
  release: {
    id: UUID;
    name: string;
    components: Array<{ name: string; digest: string }>;
  };
  environment: {
    id: UUID;
    name: string;
  };

  // Gate results
  gateResults: Array<{
    gateName: string;
    gateType: string;
    passed: boolean;
    blocking: boolean;
    message: string;
    details: object;
    evaluatedAt: DateTime;
  }>;

  // Approvals
  approvals: Array<{
    approverId: UUID;
    approverName: string;
    action: "approved" | "rejected";
    comment?: string;
    timestamp: DateTime;
  }>;

  // Context
  requester: {
    id: UUID;
    name: string;
  };
  requestReason: string;

  // Signature
  contentHash: string;
  signature: string;
}

API Endpoints

# Request promotion
POST /api/v1/promotions
Body: { releaseId, targetEnvironmentId, reason? }
Response: Promotion

# Approve/reject promotion
POST /api/v1/promotions/{id}/approve
POST /api/v1/promotions/{id}/reject
Body: { comment? }
Response: Promotion

# Cancel promotion
POST /api/v1/promotions/{id}/cancel
Response: Promotion

# Get decision record
GET /api/v1/promotions/{id}/decision
Response: DecisionRecord

# Preview gates (dry run)
POST /api/v1/promotions/preview-gates
Body: { releaseId, targetEnvironmentId }
Response: { wouldPass: boolean, gates: GateResult[] }

References

• Workflow Templates (archived)
• Workflow Execution (archived)
• Evidence Schema