Deployment Strategies

Overview

Release Orchestrator supports multiple deployment strategies to balance deployment speed, risk, and availability requirements.

Strategy Comparison

Implementation Truth Table

Strategy support must be backed by target-side runtime evidence. A single marker file or single-container replacement is not enough to claim progressive delivery.

If the traffic adapter is missing, Stella Ops may report standby slot deployment evidence, but must not report canary, blue-green, or A/B traffic-shift success. The deployment engine records this as no-adapter strategy evidence on each executed task: strategy.traffic.mode=standby_only, strategy.traffic.status=not_shifted, strategy.traffic.shifted=false, and a reason of traffic_adapter_missing or traffic_adapter_executor_missing. When the request explicitly requires traffic adapter execution, the deployment fails after target deployment evidence is captured and before any traffic-shift success is claimed.

When a traffic adapter is configured and registered, the WebApi deployment pipeline now executes it after the batch target evidence is captured and before the strategy gate advances. The adapter receives the job id, strategy, slot, variant id, batch index/count, target endpoints, and computed control/treatment weights. Canary traffic weights can be supplied as a staged list through strategyConfig.trafficSplit such as 0,25,100 or a JSON array; without an explicit split, canary starts at 5% treatment and advances by batch, while blue-green cuts over to 100% treatment after the standby batch verifies. Successful adapter execution records strategy.traffic.mode=traffic_shift, strategy.traffic.status=shifted, strategy.traffic.shifted=true, and adapter evidence. Adapter exceptions or disabled adapters fail the traffic gate. Variant/A-B style splits use the same adapter contract until a dedicated non-Kubernetes A/B strategy enum exists: strategyConfig.variantId becomes the slot/variant identity and strategyConfig.trafficSplit supplies the treatment weight sent to the adapter.

Rollback of a traffic-shifted strategy is a separate traffic-adapter operation, not just target-side Compose restore. After eligible Compose targets restore their retained revision, the pipeline calls the configured adapter in rollback mode. Blue-green rollback resolves the previous slot (green rolls back to blue, and blue rolls back to green); canary rollback resolves to active. Rollback evidence is recorded as strategy.traffic.rollback.mode=traffic_rollback, strategy.traffic.rollback.status, strategy.traffic.rollback.shifted, strategy.traffic.rollback.reason, and adapter-specific rollback fields. If requireTrafficAdapter is true, a missing or failed rollback adapter makes the rollback fail closed instead of claiming traffic was restored.

The built-in local-file adapter is an opt-in lab adapter. It writes the desired route state to a local JSON file for deterministic E2E proof of adapter execution. It is not a production router and does not replace an Nginx, HAProxy, ALB, or experiment-system adapter for real traffic.

The built-in webhook adapter is also opt-in. It posts shift and rollback requests to operator-configured URLs and records the controller response as strategy.traffic.* evidence. The Sprint 028 VM lab uses this boundary with a Playwright-owned Nginx controller: Stella Ops calls the webhook, the controller rewrites and reloads the lab router, and the test verifies the public router endpoint moves from the control upstream to the treatment upstream. That proves the Stella Ops traffic-adapter contract against a real data plane, while production-specific credentials, rollback policies, and Nginx/HAProxy/ALB controller implementations remain operator-specific adapters.

Slot evidence is mandatory for non-Kubernetes Docker/Compose targets. Strategy defaults are active for all-at-once/rolling, canary for canary/gradual, and green for blue-green/A-B standby deployment. A request or release component may override the slot with slot, deployment.slot, or a variant id. Explicit Compose project names are still made slot-specific for non-active slots, so a standby deploy cannot overwrite the active project. Target labels, Compose labels, version stickers, and rollback metadata must carry the slot and traffic role (active, standby, or canary).

Health Verification & Failure Behavior (as built)

The per-strategy HealthCheckConfig / failureBehavior TypeScript blocks further down this page are illustrative design pseudocode. The SHIPPED health/rollback behavior is the following, shared by every strategy.

• Post-deploy service health probes are configured per release component with healthProbe.* config keys (healthProbe.type=http|tcp, path/port/ expected status, retries, interval, timeout, initial delay — full key table in runtime-contract.md). The target executor runs them right after the agent deploy task succeeds, for every strategy. A probe failure fails that target’s task with a recorded reason, which the strategy gate then treats like any other target failure (rolling/canary halt before later batches). The job-level opt-out is strategyConfig.waitForHealthCheck=false, recorded as skipped in evidence.
• Automatic rollback on failure (strategyConfig.autoRollbackOnFailure, alias rollbackOnFailure, DEFAULT ON): when the deployment job fails — including via a health-probe failure — the engine makes exactly one rollback attempt of every target with retained-revision evidence, in reverse batch order, through the same compose.rollback path as operator-initiated rollback. No retained revision ⇒ the job stays failed with a recorded auto-rollback skipped reason. A failed rollback is never retried automatically. Details and evidence contract: runtime-contract.md.
• There is no metric-based auto-advance (canary metricsCheck) or validation-test stage in the shipped engine; canary/blue-green advance on target-task success + traffic-adapter evidence as described above.

All-at-Once Strategy

Description

Deploys to all targets simultaneously. Simple and fast, but highest risk.

           ALL-AT-ONCE DEPLOYMENT

  Time 0                    Time 1
  ┌─────────────────┐       ┌─────────────────┐
  │  Target 1 [v1]  │       │  Target 1 [v2]  │
  ├─────────────────┤       ├─────────────────┤
  │  Target 2 [v1]  │  ───► │  Target 2 [v2]  │
  ├─────────────────┤       ├─────────────────┤
  │  Target 3 [v1]  │       │  Target 3 [v2]  │
  └─────────────────┘       └─────────────────┘

Configuration

interface AllAtOnceConfig {
  strategy: "all-at-once";

  // Concurrency limit (0 = unlimited)
  maxConcurrent: number;

  // Health check after deployment
  healthCheck: HealthCheckConfig;

  // Failure behavior
  failureBehavior: "rollback" | "continue" | "pause";
}

// Example
const config: AllAtOnceConfig = {
  strategy: "all-at-once",
  maxConcurrent: 0,
  healthCheck: {
    type: "http",
    path: "/health",
    timeout: 30,
    retries: 3,
    interval: 10
  },
  failureBehavior: "rollback"
};

Execution

class AllAtOnceExecutor {
  async execute(job: DeploymentJob, config: AllAtOnceConfig): Promise<void> {
    const tasks = job.tasks;
    const concurrency = config.maxConcurrent || tasks.length;

    // Execute all tasks with concurrency limit
    const results = await pMap(
      tasks,
      async (task) => {
        try {
          await this.executeTask(task);
          return { taskId: task.id, success: true };
        } catch (error) {
          return { taskId: task.id, success: false, error };
        }
      },
      { concurrency }
    );

    // Check for failures
    const failures = results.filter(r => !r.success);

    if (failures.length > 0) {
      if (config.failureBehavior === "rollback") {
        await this.rollbackAll(job);
        throw new DeploymentFailedError(failures);
      } else if (config.failureBehavior === "pause") {
        job.status = "failed";
        throw new DeploymentFailedError(failures);
      }
      // "continue" - proceed despite failures
    }

    // Health check all targets
    await this.verifyAllTargets(job, config.healthCheck);
  }
}

Use Cases

• Development environments
• Small deployments
• Time-critical updates
• Stateless services with fast startup

Rolling Strategy

Description

Deploys to targets in configurable batches, maintaining availability throughout.

           ROLLING DEPLOYMENT (batch size: 1)

  Time 0            Time 1            Time 2            Time 3
  ┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
  │ T1 [v1]     │   │ T1 [v2] ✓   │   │ T1 [v2] ✓   │   │ T1 [v2] ✓   │
  ├─────────────┤   ├─────────────┤   ├─────────────┤   ├─────────────┤
  │ T2 [v1]     │──►│ T2 [v1]     │──►│ T2 [v2] ✓   │──►│ T2 [v2] ✓   │
  ├─────────────┤   ├─────────────┤   ├─────────────┤   ├─────────────┤
  │ T3 [v1]     │   │ T3 [v1]     │   │ T3 [v1]     │   │ T3 [v2] ✓   │
  └─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘

Configuration

interface RollingConfig {
  strategy: "rolling";

  // Batch configuration
  batchSize: number;           // Targets per batch
  batchPercent?: number;       // Alternative: percentage of targets

  // Timing
  batchDelay: number;          // Seconds between batches
  stabilizationTime: number;   // Wait after health check passes

  // Health check
  healthCheck: HealthCheckConfig;

  // Failure handling
  maxFailedBatches: number;    // Failures before stopping
  failureBehavior: "rollback" | "pause" | "skip";

  // Ordering
  targetOrder: "default" | "shuffle" | "priority";
}

// Example
const config: RollingConfig = {
  strategy: "rolling",
  batchSize: 2,
  batchDelay: 30,
  stabilizationTime: 60,
  healthCheck: {
    type: "http",
    path: "/health",
    timeout: 30,
    retries: 5,
    interval: 10
  },
  maxFailedBatches: 1,
  failureBehavior: "rollback",
  targetOrder: "default"
};

Execution

class RollingExecutor {
  async execute(job: DeploymentJob, config: RollingConfig): Promise<void> {
    const tasks = this.orderTasks(job.tasks, config.targetOrder);
    const batches = this.createBatches(tasks, config);
    let failedBatches = 0;
    const completedTasks: DeploymentTask[] = [];

    for (const batch of batches) {
      this.emitProgress(job, {
        phase: "deploying",
        currentBatch: batches.indexOf(batch) + 1,
        totalBatches: batches.length,
        completedTargets: completedTasks.length,
        totalTargets: tasks.length
      });

      // Execute batch
      const results = await Promise.all(
        batch.map(task => this.executeTask(task))
      );

      // Check batch results
      const failures = results.filter(r => !r.success);

      if (failures.length > 0) {
        failedBatches++;

        if (failedBatches > config.maxFailedBatches) {
          if (config.failureBehavior === "rollback") {
            await this.rollbackCompleted(completedTasks);
          }
          throw new DeploymentFailedError(failures);
        }

        if (config.failureBehavior === "pause") {
          job.status = "failed";
          throw new DeploymentFailedError(failures);
        }
        // "skip" - continue to next batch
      }

      // Health check batch targets
      await this.verifyBatch(batch, config.healthCheck);

      // Wait for stabilization
      if (config.stabilizationTime > 0) {
        await sleep(config.stabilizationTime * 1000);
      }

      completedTasks.push(...batch);

      // Wait before next batch
      if (batches.indexOf(batch) < batches.length - 1) {
        await sleep(config.batchDelay * 1000);
      }
    }
  }

  private createBatches(
    tasks: DeploymentTask[],
    config: RollingConfig
  ): DeploymentTask[][] {
    const batchSize = config.batchPercent
      ? Math.ceil(tasks.length * config.batchPercent / 100)
      : config.batchSize;

    const batches: DeploymentTask[][] = [];
    for (let i = 0; i < tasks.length; i += batchSize) {
      batches.push(tasks.slice(i, i + batchSize));
    }

    return batches;
  }
}

Use Cases

• Production deployments
• High-availability requirements
• Large target counts
• Services requiring gradual rollout

Canary Strategy

Description

Deploys to a small subset of targets first, validates, then expands to remaining targets.

           CANARY DEPLOYMENT

  Phase 1: Canary (10%)     Phase 2: Expand (50%)    Phase 3: Full (100%)

  ┌─────────────┐           ┌─────────────┐          ┌─────────────┐
  │ T1 [v2] ✓   │ ◄─canary  │ T1 [v2] ✓   │          │ T1 [v2] ✓   │
  ├─────────────┤           ├─────────────┤          ├─────────────┤
  │ T2 [v1]     │           │ T2 [v2] ✓   │          │ T2 [v2] ✓   │
  ├─────────────┤           ├─────────────┤          ├─────────────┤
  │ T3 [v1]     │           │ T3 [v2] ✓   │          │ T3 [v2] ✓   │
  ├─────────────┤           ├─────────────┤          ├─────────────┤
  │ T4 [v1]     │           │ T4 [v2] ✓   │          │ T4 [v2] ✓   │
  ├─────────────┤           ├─────────────┤          ├─────────────┤
  │ T5 [v1]     │           │ T5 [v1]     │          │ T5 [v2] ✓   │
  └─────────────┘           └─────────────┘          └─────────────┘

        │                         │                        │
        ▼                         ▼                        ▼
   Health Check              Health Check             Health Check
   Error Rate Check          Error Rate Check         Error Rate Check

Configuration

interface CanaryConfig {
  strategy: "canary";

  // Canary stages
  stages: CanaryStage[];

  // Canary selection
  canarySelector: "random" | "labeled" | "first";
  canaryLabel?: string;        // Label for canary targets

  // Automatic vs manual progression
  autoProgress: boolean;

  // Health and metrics checks
  healthCheck: HealthCheckConfig;
  metricsCheck?: MetricsCheckConfig;
}

interface CanaryStage {
  name: string;
  percentage: number;          // Target percentage
  duration: number;            // Minimum time at this stage (seconds)
  autoProgress: boolean;       // Auto-advance after duration
}

interface MetricsCheckConfig {
  integrationId: UUID;         // Metrics integration
  queries: MetricQuery[];
  failureThreshold: number;    // Percentage deviation to fail
}

interface MetricQuery {
  name: string;
  query: string;               // PromQL or similar
  operator: "lt" | "gt" | "eq";
  threshold: number;
}

// Example
const config: CanaryConfig = {
  strategy: "canary",
  stages: [
    { name: "canary", percentage: 10, duration: 300, autoProgress: false },
    { name: "expand", percentage: 50, duration: 300, autoProgress: true },
    { name: "full", percentage: 100, duration: 0, autoProgress: true }
  ],
  canarySelector: "labeled",
  canaryLabel: "canary=true",
  autoProgress: false,
  healthCheck: {
    type: "http",
    path: "/health",
    timeout: 30,
    retries: 5,
    interval: 10
  },
  metricsCheck: {
    integrationId: "prometheus-uuid",
    queries: [
      {
        name: "error_rate",
        query: "rate(http_requests_total{status=~\"5..\"}[5m]) / rate(http_requests_total[5m])",
        operator: "lt",
        threshold: 0.01  // Less than 1% error rate
      }
    ],
    failureThreshold: 10
  }
};

Execution

class CanaryExecutor {
  async execute(job: DeploymentJob, config: CanaryConfig): Promise<void> {
    const tasks = this.orderTasks(job.tasks, config);

    for (const stage of config.stages) {
      const targetCount = Math.ceil(tasks.length * stage.percentage / 100);
      const stageTasks = tasks.slice(0, targetCount);
      const newTasks = stageTasks.filter(t => t.status === "pending");

      this.emitProgress(job, {
        phase: "canary",
        stage: stage.name,
        percentage: stage.percentage,
        targets: stageTasks.length
      });

      // Deploy to new targets in this stage
      await Promise.all(newTasks.map(task => this.executeTask(task)));

      // Health check stage targets
      await this.verifyTargets(stageTasks, config.healthCheck);

      // Metrics check if configured
      if (config.metricsCheck) {
        await this.checkMetrics(stageTasks, config.metricsCheck);
      }

      // Wait for stage duration
      if (stage.duration > 0) {
        await this.waitWithMonitoring(
          stageTasks,
          stage.duration,
          config.metricsCheck
        );
      }

      // Wait for manual approval if not auto-progress
      if (!stage.autoProgress && stage.percentage < 100) {
        await this.waitForApproval(job, stage.name);
      }
    }
  }

  private async checkMetrics(
    targets: DeploymentTask[],
    config: MetricsCheckConfig
  ): Promise<void> {
    const metricsClient = await this.getMetricsClient(config.integrationId);

    for (const query of config.queries) {
      const result = await metricsClient.query(query.query);

      const passed = this.evaluateMetric(result, query);

      if (!passed) {
        throw new CanaryMetricsFailedError(query.name, result, query.threshold);
      }
    }
  }
}

Use Cases

• Risk-sensitive deployments
• Services with real user traffic
• Deployments with metrics-based validation
• Gradual feature rollouts

Blue-Green Strategy

Description

Deploys to a parallel “green” environment while “blue” continues serving traffic, then switches.

           BLUE-GREEN DEPLOYMENT

  Phase 1: Deploy Green         Phase 2: Switch Traffic

  ┌─────────────────────────┐   ┌─────────────────────────┐
  │  Load Balancer          │   │  Load Balancer          │
  │         │               │   │         │               │
  │         ▼               │   │         ▼               │
  │  ┌─────────────┐        │   │  ┌─────────────┐        │
  │  │ Blue [v1]   │◄─active│   │  │ Blue [v1]   │        │
  │  │ T1, T2, T3  │        │   │  │ T1, T2, T3  │        │
  │  └─────────────┘        │   │  └─────────────┘        │
  │                         │   │                         │
  │  ┌─────────────┐        │   │  ┌─────────────┐        │
  │  │ Green [v2]  │◄─deploy│   │  │ Green [v2]  │◄─active│
  │  │ T4, T5, T6  │        │   │  │ T4, T5, T6  │        │
  │  └─────────────┘        │   │  └─────────────┘        │
  │                         │   │                         │
  └─────────────────────────┘   └─────────────────────────┘

Configuration

interface BlueGreenConfig {
  strategy: "blue-green";

  // Environment labels
  blueLabel: string;           // Label for blue targets
  greenLabel: string;          // Label for green targets

  // Traffic routing
  routerIntegration: UUID;     // Router/LB integration
  routingConfig: RoutingConfig;

  // Validation
  healthCheck: HealthCheckConfig;
  warmupTime: number;          // Seconds to warm up green
  validationTests?: string[];  // Test suites to run

  // Switchover
  switchoverMode: "instant" | "gradual";
  gradualSteps?: number[];     // Percentage steps for gradual

  // Rollback
  keepBlueActive: number;      // Seconds to keep blue ready
}

// Example
const config: BlueGreenConfig = {
  strategy: "blue-green",
  blueLabel: "deployment=blue",
  greenLabel: "deployment=green",
  routerIntegration: "nginx-lb-uuid",
  routingConfig: {
    upstreamName: "myapp",
    healthEndpoint: "/health"
  },
  healthCheck: {
    type: "http",
    path: "/health",
    timeout: 30,
    retries: 5,
    interval: 10
  },
  warmupTime: 60,
  validationTests: ["smoke-test-suite"],
  switchoverMode: "instant",
  keepBlueActive: 1800  // 30 minutes
};

Execution

class BlueGreenExecutor {
  async execute(job: DeploymentJob, config: BlueGreenConfig): Promise<void> {
    // Identify blue and green targets
    const { blue, green } = this.categorizeTargets(job.tasks, config);

    // Phase 1: Deploy to green
    this.emitProgress(job, { phase: "deploying-green" });

    await Promise.all(green.map(task => this.executeTask(task)));

    // Health check green targets
    await this.verifyTargets(green, config.healthCheck);

    // Warmup period
    if (config.warmupTime > 0) {
      this.emitProgress(job, { phase: "warming-up" });
      await sleep(config.warmupTime * 1000);
    }

    // Run validation tests
    if (config.validationTests?.length) {
      this.emitProgress(job, { phase: "validating" });
      await this.runValidationTests(green, config.validationTests);
    }

    // Phase 2: Switch traffic
    this.emitProgress(job, { phase: "switching-traffic" });

    if (config.switchoverMode === "instant") {
      await this.instantSwitchover(config, blue, green);
    } else {
      await this.gradualSwitchover(config, blue, green);
    }

    // Verify traffic routing
    await this.verifyRouting(green, config);

    // Schedule blue decommission
    if (config.keepBlueActive > 0) {
      this.scheduleBlueDecommission(blue, config.keepBlueActive);
    }
  }

  private async instantSwitchover(
    config: BlueGreenConfig,
    blue: DeploymentTask[],
    green: DeploymentTask[]
  ): Promise<void> {
    const router = await this.getRouter(config.routerIntegration);

    // Update upstream to green targets
    await router.updateUpstream(config.routingConfig.upstreamName, {
      servers: green.map(t => ({
        address: t.target.address,
        weight: 1
      }))
    });

    // Remove blue from rotation
    await router.removeServers(
      config.routingConfig.upstreamName,
      blue.map(t => t.target.address)
    );
  }

  private async gradualSwitchover(
    config: BlueGreenConfig,
    blue: DeploymentTask[],
    green: DeploymentTask[]
  ): Promise<void> {
    const router = await this.getRouter(config.routerIntegration);
    const steps = config.gradualSteps || [25, 50, 75, 100];

    for (const percentage of steps) {
      await router.setTrafficSplit(config.routingConfig.upstreamName, {
        blue: 100 - percentage,
        green: percentage
      });

      // Monitor for errors
      await this.monitorTraffic(30);
    }
  }
}

Use Cases

• Zero-downtime deployments
• Database migration deployments
• High-stakes production updates
• Instant rollback requirements

Strategy Selection Guide

                    STRATEGY SELECTION

                         START
                           │
                           ▼
              ┌────────────────────────┐
              │  Zero downtime needed? │
              └───────────┬────────────┘
                          │
               No         │         Yes
               │          │          │
               ▼          │          ▼
         ┌──────────┐     │    ┌───────────────────┐
         │ All-at-  │     │    │ Metrics-based     │
         │ once     │     │    │ validation needed?│
         └──────────┘     │    └─────────┬─────────┘
                          │              │
                          │    No        │        Yes
                          │    │         │         │
                          │    ▼         │         ▼
                          │  ┌──────────┐│    ┌──────────┐
                          │  │ Instant   ││    │ Canary   │
                          │  │ rollback? ││    │          │
                          │  └────┬─────┘│    └──────────┘
                          │       │      │
                          │ No    │ Yes  │
                          │ │     │  │   │
                          │ ▼     │  ▼   │
                          │┌──────┐│┌────┴─────┐
                          ││Rolling│││Blue-Green│
                          │└──────┘│└──────────┘
                          │       │
                          └───────┘

References

• Deployment Overview
• Progressive Delivery
• Rollback Management