Runbook: Release Orchestrator - Rollback Operation Failed

For platform and release on-call during an incident. Use this when a Stella Ops deployment rollback will not start, returns a non-2xx, hangs in rolling_back, or lands in failed with a rollback_failed event. It explains how rollback actually works (a state transition on an existing deployment job, driven through the HTTP API — there is no live CLI path today) and how to recover safely.

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-004 - Release Orchestrator Runbooks Reconciled: 2026-05-31 against src/ReleaseOrchestrator (deployment + v2 run/environment endpoints, LibraryDeploymentRequestStarter/WebApiDeploymentRollbackStarter), src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Release, the scope catalog StellaOpsScopes.cs, and the live CLI root (Cli/StellaOps.Cli/Commands/CommandFactory.cs, Program.cs).

Metadata

FieldValue
ComponentRelease Orchestrator (src/ReleaseOrchestrator)
SeverityCritical
On-call scopePlatform team, Release team
Last updated2026-05-31
Doctor checkcheck.release.rollback.readiness (plugin stellaops.doctor.release)

How rollback works (ground truth)

A rollback is a state transition on an existing deployment job, not a standalone resource. The operation is exposed over HTTP at:

POST /api/v1/release-orchestrator/deployments/{id}/rollback
POST /api/release-orchestrator/deployments/{id}/rollback   (unversioned alias)
Body (optional): { "reason": "<free text for the audit trail>" }

(Source: __Apps/StellaOps.ReleaseOrchestrator.WebApi/Endpoints/DeploymentEndpoints.cs.)

Key facts to keep in mind while diagnosing:

NOTE (CLI surface — no live rollback command): the live stella CLI root is built by CommandFactory.Create (the entrypoint invoked from Cli/StellaOps.Cli/Program.cs), and it does not register any release rollback/release deploy command group. The release ... rollback/release deploy run|status|history handlers in Cli/StellaOps.Cli/Commands/ReleaseCommandGroup.cs (BuildReleaseCommand) print hardcoded sample output and — equally important — BuildReleaseCommand is never added to the root, so those commands are unreachable dead code. The parallel Cli/StellaOps.Cli/CliApplication.cs tree (with deploy rollback/deploy status/deploy logs against /api/v1/deployments, backed by a DeployCommandHandler whose methods return Task.CompletedTask) is likewise never instantiated as the live root and is also dead code. There is therefore no CLI path to roll back a deployment today — drive rollbacks through the HTTP API below. (Verified against CommandFactory.Create’s root registration list and Program.cs.)

Symptoms

Impact

Impact TypeDescription
User-facingRollback blocked; potentially broken release in production
Data integrityEnvironment may be in partial rollback state (rolling_back not converging)
SLA impactIncident resolution blocked; extended outage

Diagnosis

The examples below use curl against the orchestrator HTTP API because the live CLI has no rollback/deploy command (see the CLI NOTE above). Send a bearer token with the release:read scope for reads and release:publish for the rollback itself, plus the X-Tenant-Id your gateway expects.

Quick checks

  1. Check Doctor diagnostics (rollback readiness):

    stella doctor --check check.release.rollback.readiness

    This check probes the orchestrator for production environments whose rollback is blocked or that lack a previous version / health probe. (Source: RollbackReadinessCheck.cs.)

  2. Inspect the deployment job state:

    curl -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT" \
  "$ORCH/api/v1/release-orchestrator/deployments/$DEPLOYMENT_ID"

    Look at status (rolling_back, rolled_back, failed, …), progress, and canRollback.

  3. Check deployment events (rollback_started / rollback_failed):

    curl -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT" \
  "$ORCH/api/v1/release-orchestrator/deployments/$DEPLOYMENT_ID/events"

  4. Review per-environment deployment history (previous versions available to roll back to):

    curl -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT" \
  "$ORCH/api/v1/environments/$ENV_ID/deployments"

    (Source: ReleaseControlV2Endpoints.MapEnvironmentsV2 -> GET /api/v1/environments/{id}/deployments.)

    CAVEAT: this v2 route is not yet backed by live data — GetEnvironmentDeployments serves a hardcoded in-memory EnvironmentCatalog (only the demo env-production/env-staging entries), so it returns fixture deployment history rather than the calling tenant’s real deployments. For authoritative per-deployment state use the deployment detail/events endpoints above; treat this route as illustrative until it is wired to the deployment store.

Deep diagnosis

  1. Read the failure reason from logs and events. The rollback_failed event message carries the pipeline FailureReason. Pull logs for the deployment:

    curl -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT" \
  "$ORCH/api/v1/release-orchestrator/deployments/$DEPLOYMENT_ID/logs?level=error"

  2. Confirm a previous release exists to roll back to. A noPreviousVersion/canRollback=false condition is the most common blocker. Per the readiness check, the recognised causes are: previous deployment artifacts not retained, a non-reversible database migration, a breaking change deployed, or rollback manually disabled. (Source: RollbackReadinessCheck.WithCauses(...).)

  3. Check the engine path. If the deployment ID is a GUID and the real engine is enabled, the rollback runs through IDeploymentEnginePipeline. If RollbackAsync logged “no IDeploymentEnginePipeline registered” or “job not found”, the request silently fell back to a compat-store transition and no real recovery executed - check the orchestrator service logs for those warnings. (Source: WebApiDeploymentRollbackStarter.)

  4. Verify scope and tenant if the POST is rejected: 403 -> missing release:publish; tenant errors -> the route’s RequireTenant() filter rejected the request before reaching rollback logic.

Resolution

Immediate mitigation

  1. Re-issue the rollback with an audit reason (idempotent if the job is in an allowed state):

    curl -X POST -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT" \
  -H "Content-Type: application/json" \
  -d '{"reason":"<incident-id>: roll back failed release"}' \
  "$ORCH/api/v1/release-orchestrator/deployments/$DEPLOYMENT_ID/rollback"

    A 409 response means the deployment is not in completed/succeeded/failed/running/paused; resolve the current state first (e.g. cancel a stuck run) before retrying.

  2. If a run is stuck before you can roll back, cancel it, then redeploy the last known-good release as a fresh deployment:

    # cancel the in-flight deployment (requires release:write)
curl -X POST -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT" \
  "$ORCH/api/v1/release-orchestrator/deployments/$DEPLOYMENT_ID/cancel"

# create a new deployment pinning the previous (known-good) release
curl -X POST -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT" \
  -H "Content-Type: application/json" \
  -d '{"releaseId":"<previous-release-id>","environmentId":"<env-id>","strategy":"rolling"}' \
  "$ORCH/api/v1/release-orchestrator/deployments"

    (Source: DeploymentEndpoints.CreateAsync; strategy must be one of rolling, canary, blue_green, all_at_once.)

Root cause fix

If no previous version is available (canRollback=false, noPreviousVersion):

  1. The readiness check’s manual remediation is to configure artifact retention so prior releases remain rollback-eligible. The check surfaces this exact step (step 3, CommandType.Manual):

    stella config set Release:ArtifactRetention:Count 5

    (Source: RollbackReadinessCheck.WithRemediation step 3.) NB: stella config set is not a live CLI command — the live stella config group exposes only show/list/notify/integrations/feeds/registry/sources, with no generic set. Treat the surfaced command as descriptive of the intended config key; set Release:ArtifactRetention:Count through the orchestrator’s configuration (env/appsettings/DB overrides) rather than this command. (There is likewise no stella registry retention/restore command.)

  2. If the previous artifact was pruned, redeploy from a build that is still present in the registry. NB: the stella registry list / stella registry tags commands defined in Cli/StellaOps.Cli/Commands/RegistryCommandGroup.cs (BuildRegistryCommand) are not registered in the live CLI root (CommandFactory.Create only wires stella config registry for registry configuration), so there is currently no live CLI to enumerate registry contents — inspect the registry directly or through its own API/UI.

If the deploy engine pipeline is not executing the rollback:

  1. Confirm DeployEngineSwitch.UseRealEngine is enabled and an IDeploymentEnginePipeline is registered in the orchestrator. If the service logs the “no IDeploymentEnginePipeline registered” warning, the rollback only flips compat-store status and performs no real recovery.

  2. Ensure the deployment ID is the canonical GUID; non-GUID IDs bypass the real engine entirely.

If a production environment is missing a health probe (noHealthProbe warning):

  1. The readiness check recommends configuring a health probe and enabling auto-rollback-on-failure for the environment (surfaced as manual steps; there is no live stella env configure command yet). Track environment readiness via check.release.environment.readiness and check.environment.deployments.

Verification

# Verify the deployment reached a terminal rollback state
curl -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT" \
  "$ORCH/api/v1/release-orchestrator/deployments/$DEPLOYMENT_ID"
# Expect status = "rolled_back" and a rollback_completed event

# Re-run the Doctor readiness check
stella doctor --check check.release.rollback.readiness

Prevention

Reconciliation note: the original draft referenced docs/modules/release-jobengine/rollback.md and docs/operations/rollback-procedures.md; neither exists in this docs tree. Links above point at the documents that do exist.