Orchestrator Service Level Objectives

Audience: orchestrator on-call engineers and SREs operating the Stella Ops run/queue control plane.

This guide defines the service level objectives (SLOs), error-budget policy, alerts, and dashboards for the orchestrator control plane — run execution, queue health, and event delivery.

Module ownership: The generic run/queue/DAG control plane these SLOs describe is owned by the Release Orchestrator module (src/ReleaseOrchestrator/), not JobEngine. See ../../release-orchestrator/operations/metrics.mdand ../README.mdfor the JobEngine vs. Release Orchestrator boundary. This guide is filed here for historical reasons; treat the Release Orchestrator operations docs as the canonical home for run/queue telemetry.

Last updated: 2025-11-25 (DOCS-ORCH-34-005).

Service level objectives

• Availability: 99.9% monthly for the WebService API, per tenant.
• Run completion: P95 run duration < 5m for standard DAGs; failure rate < 1% over 30 days.
• Queue health: backlog < 1000 items per tenant for > 95% of 5-minute windows.
• Event delivery: WebSocket/stream delivery success > 99.5% (per day).

Error-budget policy

• Window: 28 days.
• Burn alerts:
  • 2× burn → page on-call.
  • 14× burn → immediate mitigation (disable offending DAGs, scale workers).

Alerts (examples)

• Availability: probe_success{job="orchestrator-api"} < 0.999 over 1h.
• Latency: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le,route)) > 0.5.
• Run failures: rate(orchestrator_runs_total{status="failed"}[30m]) / rate(orchestrator_runs_total[30m]) > 0.01.
• Queue backlog: orchestrator_queue_depth > 1000 for 10m.

Dashboards

• Golden signals per service (traffic, errors, latency, saturation).
• Run-outcome panel: success/fail/cancel counts, retry counts.
• Queue panel: depth, age, worker consumption rate.
• Burn-rate panel tied to the error budget.

Ownership and review

• SLOs are owned by the Orchestrator Guild and reviewed quarterly or when architecture changes.
• Changes must be reflected in the runbook and alert rules; update the manifests for offline/air-gap monitoring kits.