Runbook: Release Orchestrator - Promotion Not Progressing

For platform and release on-call. Use this when a Stella Ops release promotion appears “stuck” — the dashboard timeline stops advancing, the UI spins, or a downstream environment never receives the promoted artifact. It walks you from symptom to the actual lifecycle state the promotion is wedged in (almost always AwaitingApprovalor Deploying) and the correct way to unblock it.

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-004 - Release Orchestrator Runbooks

Reconciliation note (verify against src/ReleaseOrchestrator): This runbook was reconciled against the implemented Release Orchestrator. A promotion is the StellaOps.ReleaseOrchestrator.Promotion aggregate, whose lifecycle is the PromotionStatus enum (Pending → AwaitingApproval → Approved → Deploying → Deployed, plus terminal Rejected/Cancelled/Failed/RolledBack) enforced by PromotionStateMachine. There is no in_progress state — a promotion that “looks stuck” is almost always sitting in AwaitingApproval(no approval recorded) or Deploying(deployment running). Earlier drafts referenced a stella promotion status/trace/gates command group, stella orch status/connectivity/locks/credentials, a stella service restart command, a check.orchestrator.job-health Doctor check, and an OrchestratorPromotionStuck alert — none of those exist in source. They have been corrected to the implemented surface or flagged NOT IMPLEMENTED below.

Metadata

FieldValue
ComponentRelease Orchestrator
SeverityHigh
On-call scopePlatform team, Release team
Last updated2026-05-30
Doctor checkcheck.release.active (Release Pipeline plugin; default severity Warn)

The Doctor check above is the real “stuck release” diagnostic — ActiveReleaseHealthCheck (src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Release/Checks/ActiveReleaseHealthCheck.cs). It queries the real Release Orchestrator deployment and approval read models and flags pending/running/paused/rolling_back deployments or stale pending approvals. There is no check.orchestrator.job-health check in source. The full Release Pipeline plugin also ships check.release.promotion.gates, check.release.rollback.readiness, check.release.environment.readiness, and check.release.environment.matches-approved.

Symptoms

NOT IMPLEMENTED: there is no OrchestratorPromotionStuck alert in source. The release health signal is the Doctor check.release.active result, whose thresholds are encoded in ActiveReleaseHealthCheck: an executing/pending release warns at 1h and fails at 4h; a pending_approval release warns at 4h and fails at 24h. Verify any dashboard alert is wired to those Doctor results, not to a nonexistent metric.

Impact

Impact TypeDescription
User-facingRelease blocked; cannot promote to the target environment
Data integrityArtifact is safe; the promotion is re-evaluable. A Failed promotion may transition to RolledBack or back to AwaitingApproval (the only non-terminal exits the state machine allows from Failed)
SLA impactRelease SLO violated if not resolved promptly

Diagnosis

Quick checks

  1. Run the Doctor active-release check:

    stella doctor --check check.release.active

    The evidence block reports stuck_release_count, failed_release_count, pending_approval_count, and per-release summaries (name:state:Nmin). A connection_error_type=timeout or Cannot reach Release Orchestrator Warn points at an unreachable/slow orchestrator rather than a stuck promotion.

  2. List deployments and find the stuck one:

    stella deployment list --environment <environment>
stella deployment list --status running,failed --json

    Correction, 2026-07-28. This step previously read stella promote list --env … and asserted that stella promote {start|status|approve|reject|list} “is the implemented promotion CLI”, citing src/Cli/StellaOps.Cli/Commands/PromoteCommandHandler.cs. All of that was wrong: stella promote exits 2 (Unrecognized command or argument 'promote'), and no PromoteCommandHandler.cs exists under src/Cli/ — the only file of that name in the repository is an archived audit note under docs-archive/.

    The shipped split is: stella release … acts, stella deployment … observes. There is no CLI command that lists promotions as objects; use the Console or GET /api/v1/release-orchestrator/…. The surrounding claim that stella promotion builds promotion attestations rather than lifecycle operations is correct — its verbs are assemble, attest, verify.

  3. Inspect the deployment’s current status and detail:

    stella deployment show <deployment-id>
stella deployment watch <deployment-id> --interval 5   # poll to a terminal state
stella deployment events <deployment-id>

    These print status, per-step deployment progress, and the event stream, and tell you whether the release is waiting on an approval or mid-deployment.

Deep diagnosis

  1. Determine which lifecycle state the promotion is in. Map the symptom to the state machine (PromotionStateMachine.ValidTransitions):

    • AwaitingApproval→ waiting for the required ApprovalRecord(s). Valid exits: Approved, Rejected, Cancelled. This is the dominant “stuck” case.
    • Approved→ ready to deploy. Valid exits: Deploying, Cancelled.
    • Deploying→ deployment running. Valid exits: Deployed, Failed.
    • Failed→ valid exits: RolledBack, AwaitingApproval (re-submit). A promotion in Pending was created but never submitted for approval.

  2. Check the promotion gates. Gate evaluation results live on the promotion (Promotion.GateResults, each a GateResult with GateName, GateType, Passed, Blocking). The seven built-in gates registered by GateRegistry.RegisterBuiltIn are approval-gate, security-gate, policy-gate, dependency-gate, schedule-gate, freeze-window-gate, and manual-gate. Six of the provider classes live under …/Promotion/Gate/BuiltIn/ (ApprovalGate, PolicyGate, DependencyGate, ScheduleGate, FreezeWindowGate, ManualGate); SecurityGate lives under …/Promotion/Gate/Security/. For gate health (missing policies, unreachable attestor, missing approvers) run:

    stella doctor --check check.release.promotion.gates

    See orchestrator-gate-timeout.md for the CVE-aware policy-gate timeout path (HttpPolicyEngineGateClient, fail-closed deny).

  3. Inspect gate status for the underlying release.

    REMOVED (2026-07-03): the stella release gates {list|status|approve|reject} group emitted hardcoded sample output and was deleted from the CLI (stella release now carries only the real approve/reject/promote/deploy/rollback verbs). The authoritative gate state for a promotion is Promotion.GateResults (see step 2), the Doctor check.release.promotion.gates result, and the UI Gate policies / promotion decision cockpit (/releases/promotions/<id>).

  4. Read the orchestrator logs. Look for the deployment-step and gate-evaluation lines. Gate metrics emitted by ReleaseOrchestratorPluginMonitor are stellaops_gate_evaluations_total and stellaops_gate_evaluation_duration_ms (verify the exposed names on your build).

NOT IMPLEMENTED: there is no stella orch connectivity --target <env>, no stella orch locks list, and no lock-contention surface for promotions in source. The stella orch command group operates the Source & Job Orchestrator (sources/backfill/quotas) and is unrelated to release promotions. Environment connectivity is exercised via the deployment agents (SSH/WinRM/Docker/Compose/etc.), not a promotion-level lock manager.

Resolution

Immediate mitigation

  1. If the promotion is waiting in AwaitingApproval, record the approval (or rejection). These operations require the release:publish scope (policy ReleaseOrchestratorPolicies.ReleaseApprove):

    stella release approve <approval-id> --comment "Approved by <approver>"
# or
stella release reject <approval-id> --reason "<why>"

    Note the argument is the approval request id, not the promotion id. Both verbs also accept --sign (plus --key/--key-id/--fingerprint/--algorithm) to submit the decision as an operator-decision DSSE envelope.

    REST equivalents: POST /api/v1/release-orchestrator/promotions/{id}/approve and …/reject (ReleaseDashboardEndpoints, also mirrored under /api/release-orchestrator). The v2 decision surface is POST /api/v1/approvals/{id}/decision (ReleaseControlV2Endpoints, actions approve|reject|defer|escalate).

  2. If a blocking gate is wedged, drive the decision through the promotion approval surface (CLI stella release approve|reject <approval-id>, REST …/promotions/{id}/approve or v2 …/approvals/{id}/decision) rather than a “skip” command.

    The dedicated scope to bypass release policy gates is release:bypass (StellaOpsScopes.ReleaseBypass); confirm which surface honors it before relying on a skip flow. There is no stella promotion gate skip command in source, and the former stubbed stella release gates approve / stella release promote --force sample-output commands were removed from the CLI on 2026-07-03 — drive overrides through the promotion approval surface above, never a CLI “skip”.

  3. If the promotion is Deploying and the deployment is wedged, triage the deployment itself (agent reachability, target health). A failed deployment moves the promotion to Failed; from there the state machine allows RolledBack or re-submission via AwaitingApproval. Rollback of a run goes through POST /api/v1/runs/{id}/rollback (ReleaseControlV2Endpoints), which is only accepted when the run status is failed/warning/degraded. See orchestrator-rollback-failed.md.

  4. If the orchestrator service is unresponsive, restart/redeploy the release-orchestrator container through your normal deployment tooling (compose/agent). There is no stella service restart orchestrator command in source.

Root cause fix

Approval bottleneck (promotion sits in AwaitingApproval):

  1. Confirm approvers are configured for the gate — Doctor reports no_approvers when an approval gate has none (check.release.promotion.gates).
  2. Ensure the approving principal holds release:publish; without it the approve/reject endpoints return 403.
  3. Separation-of-duties is enforced (SeparationOfDutiesEnforcer) — the requester may be blocked from self-approving. Use a different approver.

Policy gate denying or timing out:

  1. Triage with orchestrator-gate-timeout.md (CVE-aware policy gate; ReleaseOrchestrator:Policy:TimeoutSeconds, default 10s, fail-closed on timeout).
  2. Verify the environment policy binding exists (release_orchestrator.environment_policy_bindings).

Target environment unreachable (promotion sits in Deploying):

  1. Verify the deployment agent for the target environment is healthy and reachable.
  2. Check target credentials/secrets resolution (the orchestrator resolves tenant secrets at deploy time; a resolution failure surfaces as a failed deployment step).

NOT IMPLEMENTED in source (forward-looking / roadmap only — do not run): per-gate timeout retry config (stella orch config set gates.<name>.retries), optimistic locking modes (stella orch config set locks.mode optimistic), lock timeout config (stella orch config set locks.timeout), and stella orch credentials verify. The CVE gate timeout is configured via ReleaseOrchestrator:Policy:TimeoutSeconds (see the gate runbook), not a gates.<name>.timeout key.

Verification

# Re-check active releases (stuck/approval-pending counts should clear)
stella doctor --check check.release.active

# Confirm the deployment advanced
stella deployment show <deployment-id>

# Confirm gate health
stella doctor --check check.release.promotion.gates

After approval (or restoring the target / policy engine), confirm the promotion transitions through Approved → Deploying → Deployed. stella deployment watch <deployment-id> polls to a terminal state (--interval seconds, default 2). Watching is a separate verb here, not a --watch flag — stella deployment show --watch is not valid. (stella doctor does have a --watch flag; the two are unrelated.)

Prevention