Docker/Compose Runtime Contract

This contract defines when Stella Ops may report a Docker or Docker Compose deployment as successful on a non-Kubernetes target.

Success Boundary

A Docker/Compose deployment is successful only when all of these are true:

Marker files, shell command output, API status, or target connectivity are useful evidence, but they do not satisfy this success boundary by themselves.

Control-Plane Progress Projection

The deployment-monitor compatibility read model is a projection of the typed deployment job; it is not an independent execution state machine. A newly accepted deployment is first persisted as pending. While the library engine runs, the WebApi deployment scope re-reads release_orchestrator.deployment_jobs and, when that state changes, mirrors observed job status plus per-target status/progress into release_orchestrator.deployments. It must not invent a running transition or percentage that the typed job has not persisted. Terminal projection remains a separate step because terminal strategy evidence and target-scoped logs are available only after execution finishes.

The deployment scope waits for the job for at most 12 hours (LibraryDeploymentRequestStarter.DeploymentScopeTimeout); the in-flight mirror runs only while it waits. A record that is still non-terminal after that keeps its last mirrored status and target progress; the projection does not close it by elapsed time. Readers report it as recorded: the console shows the latest recorded progress time and marks a non-terminal record without progress for more than 12 hours as stale, without inferring completion or failure. The deployment record’s targets are the environment’s targets recorded when the deployment started; later topology changes do not rewrite them.

Runtime Modes

ModeBehavior
preinstalledStella Ops verifies Docker/Compose are already present and new enough. Missing or old runtimes fail before deployment.
managedStella Ops may install or upgrade Docker/Compose from an explicit package source, local mirror, or Offline Kit path.
detect-onlyStella Ops reports readiness and blockers without changing the target.

Default version gates for planning:

Production and offline deployments must not silently install from the public internet. A managed install must record the configured package source type in evidence without leaking credentials.

Preflight Evidence

Preflight runs before image pulls, Compose writes, or deployment hooks with side effects. It records:

Stable failure codes include:

Target Readiness Checks

Target health checks must validate the execution path that Stella Ops will use for the deployment. Ssh and WinRM targets use direct endpoint probes even when an agent id is present, because their tasks execute through the remote transport. Agent-bound DockerHost, ComposeHost, ECS, and Nomad targets must fail closed unless the referenced agent exists, is active, has a fresh heartbeat, and advertises the capabilities required for that target kind.

Database migrations must preserve the health status contract on existing installations. The allowed statuses are unknown, healthy, degraded, unhealthy, offline, and unreachable; legacy constraints must be refreshed in place during startup migration.

Post-Deploy Service Health Verification (as built)

Agent task success proves the deploy commands ran (container started under the compose runtime contract); it does not prove the SERVICE answers. Components opt in to post-deploy verification with healthProbe.* config keys, and the target executor runs the configured probes immediately after the agent deploy task succeeds. A probe failure fails that target’s deployment task with a recorded per-target reason (Post-deploy health probe failed for component '<name>': … in the task error / deployment_json targets[].error), which feeds the strategy gate and the final deployment status like any other target failure.

Supported component config keys (the healthProbe. prefix is distinct from healthcheck., which configures the container’s own docker healthcheck in the compose lock):

KeyMeaningDefault
healthProbe.typehttp or tcp; presence enables the probe— (opt-in)
healthProbe.http.urlabsolute probe URL (overrides host/port/path/scheme)—
healthProbe.http.pathpath on the target host/
healthProbe.http.portport80 (http) / 443 (https)
healthProbe.http.schemehttp or httpshttp
healthProbe.http.expectedStatusexact status; unset = any 200–399unset
healthProbe.tcp.portTCP connect port (required for tcp)—
healthProbe.hostprobe host overridetarget connection host
healthProbe.retriestotal attempts (min 1)3
healthProbe.intervalSecondsdelay between attempts5
healthProbe.timeoutSecondsper-attempt timeout5
healthProbe.initialDelaySecondswarm-up delay before first attempt0

Semantics (all fail-closed):

Probe placement (deliberate, revisit-with-transport-rev): probes execute ORCHESTRATOR-SIDE via IDeploymentHealthProbeRunner (default HttpTcpDeploymentHealthProbeRunner — HTTP GET / TCP connect from the control plane). Agent-side probe tasks are the preferred long-term seam for targets behind firewalls, but the DeploymentAgentTask transport contract currently has no task type that expresses a service probe (compose_preflight is host runtime readiness, not service health), and adding one requires the in-process transport, the mTLS polling transport, and remote Agent.Core binaries to rev in lockstep. The runner interface is transport-agnostic so an agent-side implementation can replace the direct prober without executor changes.

Generated Artifacts

The minimum Compose deployment artifact set is:

compose.stella.lock.yml is desired state. It must be deterministic for the same release, target, environment, and slot. Runtime timestamps belong in evidence or stickers, not in the lock file when they would change the desired state hash.

The top-level Compose version value, when present for compatibility with older Compose readers, must be emitted as a YAML string such as version: "3.8". Docker Compose v2.40 rejects an unquoted numeric value with version must be a string.

Agent Task Payload Contract

Container deployment tasks must preserve the full registry/repository image identity and the immutable digest. The runtime bridge must never collapse a component to componentName@digest when the release supplied registry/repository:tag@digest.

External polling assignments deliver the payload string directly to Agent.Core task handlers. Those handlers deserialize with the default System.Text.Json options, so task payload keys must match the handler property names exactly: Image, Name, and PullImage for docker.run; ProjectName, ComposeLock, VersionSticker, PullImages, and BackupExisting for compose.up; Hooks for docker.run and compose.up; ProjectName, optional RevisionPath and optional VerifyTimeoutSeconds (default 60) for compose.rollback; and WorkDirectory, ImageReference, MinimumFreeDiskBytes, RegistryReachabilityMode, and RequireCompose for compose.preflight. DockerHost preflight sets RequireCompose=false by default; ComposeHost sets it to true.

Hook entries are target-side task payload data, not control-plane-only labels. Each entry uses exact property names: Name, Phase, Interpreter, Command, WorkingDirectory, TimeoutSeconds, FailurePolicy, and RedactionPolicy. Supported deployment hook phases are pre-deploy and post-deploy. FailurePolicy=fail is the default and blocks the deployment; ignore, continue, and warn record the hook result without failing the phase. Hook stdout/stderr are redacted by default unless the hook explicitly sets RedactionPolicy=none.

For DockerHost, the agent task carries one digest-pinned image reference, component config, Stella labels, project/slot variables, work directory, and manifest hash. The Docker payload labels must include Stella target identity: stella.target.id, stella.target.name, stella.target.type, stella.target.endpoint, stella.execution.mode, and stella.manifest.hash. Component config keys prefixed with env. are copied into the docker.run payload environment without the prefix, alongside the standard Stella deployment variables. DockerHost payloads request PullImage=true unless the release explicitly marks a preloaded/offline image path with runtime.skipPull, compose.skipPull, compose.pullImages=false, or pullImages=false; when pull is enabled, the agent must pull and verify the digest-pinned image before listing, stopping, or creating containers.

For ComposeHost, the agent task carries:

The task payload may be delivered to a target-resident agent or to a lab bridge that explicitly executes on the target VM. It must not run Docker/Compose on the orchestrator host while reporting target-side deployment success.

Agent task results must echo the target identity and execution mode that were used for the run. Docker result evidence includes the container id/name/state, IP address, targetId, targetName, targetType, targetEndpoint, targetExecutionMode, manifestHash, completedPhases, phaseDurationsMs, deploymentPhases, hookResults, and failurePhase. Compose result evidence includes the project directory, compose file path, apply manifest path, artifact hashes, service count/list, completedPhases, phaseDurationsMs, deploymentPhases, hookResults, failurePhase, and the same target identity fields. Successful Compose deployment also returns previousRevisionPath when a rollback baseline was retained. A missing Docker/Compose in-process capability is a failure, not permission to run against the orchestrator host.

The target agent exposes compose.rollback for retained revision restoration. It resolves an explicit revision path, or the latest retained revision, restores compose.stella.lock.yml, stella.version.json, and apply-manifest.json under the project lock, validates hashes, pulls rollback images, starts Compose, and reads the restored services back with the rule compose.up applies: every service running, healthy when it reports a health check, or a successful one-shot, within VerifyTimeoutSeconds. Only then does it record rollbackRestored, runtimeVerified, restoredRevisionPath, the service list, phase timings, and target identity. A restored service that is exited or unhealthy fails the task at the compose_verify phase, and a docker compose ps that cannot run fails it at compose_ps; the restored files stay on disk and the target is not reported as recovered. If no retained revision exists, the task fails with a stable resolve_revision failure phase before any pull or runtime mutation; a named revision that was not retained fails at restore_revision before any runtime command.

Target Layout

The target project directory is owned by Stella Ops. The implementation may vary by OS, but each target must expose these logical paths:

Logical artifactPurpose
active compose lockCurrent desired Compose model
version stickerCurrent applied release metadata
apply manifestCommit marker and artifact hashes
revision storePrevious successful revisions for rollback
lock filePer-project deployment lock

Compose project names must be stable for the tuple tenant/environment/target/application/slot. They must not be random deployment ids. Valid slots include active, canary, blue, green, and A/B variant ids. If a release supplies an explicit Compose project, Stella Ops appends the non-active slot unless the project name already carries that slot suffix. This prevents a blue, green, canary, or variant deployment from overwriting the active Compose project by reusing the same explicit project name.

Slot identity is deployment evidence. Target-agent Docker and Compose assignments carry slot, trafficRole, and projectName variables. Docker container labels include stella.slot, stella.project, and stella.traffic.role; Compose labels include the same values through the generated lock. Version stickers record both slot and trafficRole. Default slot selection is strategy-aware: rolling/all-at-once use active, canary uses canary, and blue-green/A-B use green unless the release or request strategy config supplies slot, deployment.slot, or variantId.

Deployment Phases

Docker/Compose target execution uses these phases:

  1. preflight
  2. prepare
  3. pre-deploy
  4. deploy
  5. verify
  6. promote
  7. post-deploy
  8. evidence
  9. cleanup

Target task handlers also expose lower-level implementation phases in completedPhases, such as pull_image, prepare_container, pre_deploy_hooks, write_artifacts, compose_pull, compose_up, inspect_services, write_sticker, and post_deploy_hooks. These names are for diagnostics and tests; the high-level deploymentPhases list remains the stable product evidence surface.

Rollback uses the same executor with previous successful artifacts and records:

  1. rollback-pre
  2. rollback-deploy
  3. rollback-verify
  4. rollback-post

Hooks may run in phase-specific points, but hooks are not a substitute for Docker/Compose execution.

Release Script Graph Proof

Release script dependency evidence uses release.script.graph-proof.v1. The proof is generated from explicit script/module nodes and must remain deterministic for the same release graph:

The proof fails closed with status=failed when a dependency is missing, a declared dependency version does not match an available script node, duplicate nodes exist, a cycle is detected, or a start-phase script depends on an update-phase script. Update-phase scripts may depend on start-phase scripts; those cross-phase references are preserved in evidence while update ordering is computed from update-to-update dependencies. A failed graph proof is blocking assurance evidence, not a warning.

Strategy Evidence Boundary

Deployment strategy planning is evidence only when the persisted job records the applied strategy, batch indexes, target order, and continuation decision after each batch. Rolling and canary plans must stop further batches when the strategy gate says not to proceed. Unrun tasks after a failed strategy gate must be recorded as Skipped, not left as Pending.

Rollback success requires target-side restoration evidence. A deployment status transition into rolling_back proves the control-plane mutation was accepted, but it is not sufficient evidence that the previous Compose lock, image digest, labels, health, or version sticker were restored.

Automatic rollback on failed deployment (as built)

When a deployment job FAILS — including a post-deploy health-probe failure — the deploy engine (DeployOrchestrator forward path) makes exactly ONE automatic rollback attempt through the same per-target rollback path the operator-initiated pipeline uses (TargetExecutor.RollbackTargetAsync → compose.rollback on the target agent). Behavior:

When the WebApi real deployment engine is enabled, the rollback mutation must dispatch rollback work through the production deployment pipeline, not only update compatibility-store status. The pipeline selects eligible Compose target tasks with retained revision evidence, resets them for rollback, executes rollback batches in reverse batch order, and asks the target executor to publish compose.rollback assignments carrying projectName, target identity, optional rollback.revisionPath, and optional rollback.reason. Targets without Compose revision evidence are skipped with an explicit no-revision reason; if no Compose task is eligible, the rollback fails closed.

The operator rollback keeps the failure it recovers from. The job keeps its FailureReason while it rolls back and after it ends RolledBack; when the rollback itself fails, the reason reads <original reason>; rollback failed: <rollback failure>. Each target that had failed keeps its own error under rollback.originalFailure, whether the rollback restores it, fails on it, or skips it for missing revision evidence. A failure that an earlier rollback already kept on the target (for example after a failed automatic rollback) stays the original failure.

Once the operator rollback has moved the deployment to rolling_back, it runs to a terminal state independently of the HTTP request that started it: a client disconnect or gateway timeout does not cancel the rollback, its evidence packet or its read projection, so an aborted request cannot leave the job RollingBack (the target retry follows the same rule).

The deployment read (GET /api/v1/release-orchestrator/deployments/{id}) returns that kept failure on each target as originalFailure, for a restored, failed or skipped rollback target of either rollback path. error is the rollback error of a target whose rollback failed. For a restored target of a rolled_back deployment, error is the original failure on both paths; the operator path now keeps that failure, so its restored targets show it in error as the automatic path’s already did (previously null). originalFailure is null when no rollback replaced a failure on the target.

Blue-green, canary/gradual, and A/B releases require traffic adapter evidence before Stella Ops may report traffic-shifted success. Without a traffic adapter, Stella Ops may only report standby-slot deployment evidence or fail with a stable no-adapter code; it must not imply traffic moved. Current no-adapter evidence uses strategy.traffic.mode=standby_only, strategy.traffic.status=not_shifted, strategy.traffic.shifted=false, and strategy.traffic.reason=traffic_adapter_missing or traffic_adapter_executor_missing.

If strategyConfig.trafficAdapter resolves to a registered adapter, the WebApi pipeline executes the adapter after target-side deployment evidence exists and before the strategy gate advances. Adapter evidence is projected back to the deployment record under strategyEvidence, including strategy.traffic.mode, strategy.traffic.status, strategy.traffic.shifted, strategy.traffic.controlWeight, strategy.traffic.treatmentWeight, strategy.traffic.slot, and adapter-specific fields. requireTrafficAdapter turns any missing, disabled, or failed adapter into a failed deployment before later batches are started. The opt-in local-file adapter writes route state to disk for repeatable lab verification only; production traffic shifting still requires an actual router/load-balancer adapter.

The opt-in webhook adapter is the generic router-controller boundary. It is disabled unless configured, posts shift and rollback requests to the configured URLs, fails closed on missing URL, timeout, unreachable controller, non-2xx response, or invalid response, and records controller evidence such as adapter kind, webhook URL/status, observed data-plane URL, and applied timestamp. The VM lab proves this boundary with a Playwright-managed Nginx router, but production traffic control still belongs in an operator-owned controller/plugin with the appropriate router credentials and rollback policy.

Traffic rollback uses the same adapter boundary. Once target-side compose.rollback work succeeds, a traffic strategy with adapter configuration must ask the adapter to restore the previous traffic slot and then record strategy.traffic.rollback.mode=traffic_rollback, strategy.traffic.rollback.status, strategy.traffic.rollback.shifted, strategy.traffic.rollback.reason, strategy.traffic.rollback.slot, and any adapter-specific rollback evidence. A required but missing/failed adapter fails the rollback closed with stable evidence such as traffic_adapter_executor_missing or traffic_adapter_rollback_failed.

Active-deployment guard

DeployOrchestrator.StartAsync refuses a new deployment in an environment (DeploymentAlreadyInProgressException, naming the holding job) while that environment has a Pending, Running or RollingBack job. Both job stores apply the same rule: the PostgreSQL store reads release_orchestrator.deployment_jobs.status IN ('Pending', 'Running', 'RollingBack') and the in-memory store the same three states. A rollback that is restoring an environment’s targets, automatic or operator-started, therefore blocks a new deployment there until it ends RolledBack or Failed; terminal jobs (Completed, Failed, Cancelled, RolledBack) do not count.

The refusal reaches HTTP callers that start a deployment in that environment: POST /api/v1/release-orchestrator/deployments, POST /api/v1/release-orchestrator/releases/{id}/deploy and POST /api/v1/release-orchestrator/releases/seal fail while a Pending, Running or RollingBack job holds the environment. The exception is not mapped to a problem response, so the caller receives HTTP 500, and the durable deployment dispatcher can end a dispatch dispatch_failed.

A RollingBack job cannot be cancelled through the deployment API (cancellation accepts Pending and Running jobs only). An aborted operator rollback request no longer strands a job (see above). A Release Orchestrator restart during a rollback is settled at startup, as described below.

Rollback interrupted by a restart

A rollback, automatic or operator-started, runs inside the Release Orchestrator process that started it: the pipeline holds the job, and the agent task transport keeps its pending assignments in process memory. Nothing resumes a rollback after a restart, so a job that is still RollingBack when the service starts has no rollback running. While the service starts, after the startup migrations and before the deployment dispatcher and the HTTP server accept work, it settles every such job (DeploymentRollbackRestartRecovery):

Jobs in any other state, including a Running deployment, are not changed. A failure of the recovery is logged and does not stop the service; the jobs it could not settle keep holding their environments until the next start.

This recovery relies on one Release Orchestrator instance running the deploy engine. The engine has no lease, leader election or job ownership, the agent task transport serves agent polls and results only from the process that queued the task, and the shipped Compose definitions run the service as one container with a fixed name. Run the Release Orchestrator WebApi as exactly one instance per database: several replicas are not a supported shape, and a second instance starting while the first is rolling back would settle that rollback under it.

Completion Evidence

A passing deployment record must include:

If rollback succeeds after a failed deployment, the original deployment remains failed. Rollback success is recorded as a separate recovery result.

Rollback evidence generated by the deployment library uses evidenceVersion=release.rollback.evidence.v1. It records proofStatus as complete only when every rollback target has both the current digest and the rollback digest. Missing digest proof is not hidden: each target records hasDigestProof=false and a stable missingProofReason, and the top-level proofStatus becomes incomplete. The proofDigest is a deterministic SHA-256 digest over the rollback proof fields so replay and audit tooling can detect proof drift without depending on the evidence creation timestamp. The WebApi rollback starter persists this payload to release-truth evidence as release.rollback.evidence.v1 when a real rollback pipeline reaches RolledBack, and the deploy engine’s automatic rollback persists the same payload through the same writer when it restores its targets. The deployment projection carries rollback.evidence.id, rollback.evidence.hash, rollback.evidence.proofDigest, rollback.evidence.proofStatus, rollback.evidence.orderProofDigest, and rollback.evidence.orderProofStatus in strategy evidence so UI, audit, and replay tooling can navigate from the rollback result to the stored evidence packet.

The same rollback evidence also records order proof for multi-module releases. deploymentOrder[] captures the original failed job’s task/module order, including module name, target identity, batch index, timestamps, status, dependency names, failure reason, and release-script proof digest when carried by the task result. rollbackOrder[] captures the terminal rollback job’s task/module order using the same shape. failure identifies the failed module that triggered rollback. orderProofStatus=complete requires non-empty deployment and rollback order plus a failed module; otherwise it is incomplete. orderProofDigest is a deterministic SHA-256 digest over these order/failure fields. This is a replay/audit proof, not a substitute for the live target-side rollback evidence required above.