Orchestrator Runbook (DOCS-ORCH-34-003)

Last updated: 2026-05-30

Audience: operators running the Stella Ops Orchestrator (scheduling and run execution). Purpose: pre-flight checks, day-to-day run/schedule operations, incident response, health/metrics, and offline/air-gap procedures for the Orchestrator surface.

Implementation note: the “Orchestrator” is delivered by the JobEngine module, whose web service is StellaOps.Scheduler.WebService (service/router name scheduler). Operational HTTP routes are therefore served under /api/v1/scheduler/... for runs/schedules and under /api/v1/jobengine/... for the read-only Console compatibility surface. By default the web service runs in embedded worker mode (Scheduler:Worker:Embedded=true), so all six heavy worker background services (Planner, PlannerQueueDispatcher, Runner, PolicyRunDispatch, GraphBuild, GraphOverlay) run in-process; set it to false only when scaling those workers in a separate scheduler-worker host. The exception lifecycle workers (ExceptionLifecycleWorker, ExpiringNotificationWorker) and the SystemScheduleBootstrap always run in the web process regardless of the embedded flag. (Source: Program.cs — AddSchedulerWorker vs. the unconditional AddHostedService registrations.)

Pre-flight

• Ensure PostgreSQL reachable and a queue transport (Redis or NATS) configured; liveness at GET /healthz and readiness at GET /readyz return HTTP 200 ({ "status": "ok" } / { "status": "ready" }, both anonymous). There is no aggregate /admin/health endpoint and the probes do not report queue depth — use GET /api/v1/scheduler/runs/queue/lag for depth.
• Verify tenant allowlist and scopes configured in Authority. The orchestrator authorization scopes are orch:read, orch:operate, orch:quota, and orch:backfill; the underlying scheduler surface also accepts scheduler:read, scheduler:operate, and scheduler:admin. (There is no orchestrator:* scope.) Source symbols: StellaOpsScopes.OrchRead/OrchOperate/OrchQuota/OrchBackfill, StellaOpsScopes.SchedulerRead/SchedulerOperate/SchedulerAdmin.
• Two-layer scope enforcement on run/schedule endpoints (important — easy to under-provision). Each run and schedule endpoint applies two checks: (1) an ASP.NET authorization policy mapped to the coarse scope — SchedulerPolicies.Read → scheduler:read, SchedulerPolicies.Operate → scheduler:operate, SchedulerPolicies.Admin → scheduler:admin; and (2) an in-handler IScopeAuthorizer.EnsureScope check for a fine-grained scope: scheduler.runs.read/.write/.manage/.preview and scheduler.schedules.read/.write. In production (TokenScopeAuthorizer, active when Scheduler:Authority:Enabled=true) the fine-grained scope must be present literally in the token — there is no hierarchical fallback, so a token carrying only scheduler:operate is rejected at the fine-grained gate. In Development/Testing header-fallback mode (HeaderScopeAuthorizer) the coarse scope does satisfy the fine-grained gate (scheduler.<svc>.<read|write|manage|preview> is allowed by scheduler:<read|operate|admin>). Issue Authority client scopes accordingly: a production operator token for run/schedule mutations needs both scheduler:operate and the relevant scheduler.runs.*/ scheduler.schedules.* scopes. (Source: RunEndpoints/ScheduleEndpoints EnsureScope constants; TokenScopeAuthorizer vs. HeaderScopeAuthorizer; policies in Program.cs.) These dotted fine-grained scopes are not declared in the canonical StellaOpsScopes.cs catalog — they are literal strings enforced only by the scheduler handlers, so Authority client config must grant them explicitly. The JobEngine compatibility endpoints (/api/v1/jobengine/...) are the exception: they require only the coarse scheduler:read or orch:read and have no additional fine-grained gate.
• Storage connection string must be configured outside local-harness mode — resolved from the first of Scheduler:Storage:ConnectionString, Scheduler:Storage:Postgres:Scheduler:ConnectionString (compose-nested), or Postgres:Scheduler:ConnectionString (legacy) — the service refuses to start otherwise. (Source: SchedulerStorageConfiguration.)
• Plugin bundles present and signatures verified (scheduler job plugins are discovered from plugin DLLs at startup via PluginHost.LoadPlugins).

Common operations

• Create a run: POST /api/v1/scheduler/runs/ (policy scheduler:operate + fine-grained scheduler.runs.manage). The body must reference an existing scheduleId (404 if it does not exist); only trigger: manual runs may be created through this endpoint. (Source: RunEndpoints.CreateRunAsync.)
• Cancel a run: POST /api/v1/scheduler/runs/{runId}/cancel (policy scheduler:operate + fine-grained scheduler.runs.write). Best-effort; returns HTTP 409 if the run is already terminal. (Source: RunEndpoints.CancelRunAsync.)
• Retry a run: POST /api/v1/scheduler/runs/{runId}/retry (policy scheduler:operate + fine-grained scheduler.runs.manage). Creates a new manual run with retryOf pointing at the original; the original must be terminal.
• Stream status: GET /api/v1/scheduler/runs/{runId}/stream — Server-Sent Events (text/event-stream), not WebSocket. Emits initial, queueLag, heartbeat, stateChanged, segmentProgress, deltaSummary, and completed events (plus a notFound event if the run row disappears mid-stream). (Source: RunStreamCoordinator.)
• List runs / read state: GET /api/v1/scheduler/runs/ (filters: scheduleId, state, createdAfter, limit, cursor, sort), GET /api/v1/scheduler/runs/{runId}, GET /api/v1/scheduler/runs/{runId}/deltas — all read endpoints apply policy scheduler:read + fine-grained scheduler.runs.read.
• Preview impact: POST /api/v1/scheduler/runs/preview (policy scheduler:operate + fine-grained scheduler.runs.preview). Resolves the impacted image set for a scheduleId or inline selector — optionally narrowed by productKeys/vulnerabilityIds and usageOnly — and returns a bounded sample (sampleSize clamped to 1–50). Read-only; does not create a run. (Source: RunEndpoints.PreviewImpactAsync.)
• Manage schedules: under /api/v1/scheduler/schedules — list (GET /), read (GET /{scheduleId}), create (POST /), update (PATCH /{scheduleId}), soft-delete (DELETE /{scheduleId}; system-managed schedules cannot be deleted), and pause/resume (POST /{scheduleId}/pause and /resume). List/read apply policy scheduler:read + fine-grained scheduler.schedules.read; all mutations (create/update/delete/pause/resume) apply policy scheduler:operate + fine-grained scheduler.schedules.write. Pausing a schedule is the direct lever to stop its scheduled runs (distinct from stella orch sources pause, which pauses a source). (Source: ScheduleEndpoints.)
• Console job view (read-only): GET /api/v1/jobengine/jobs, /api/v1/jobengine/jobs/{jobId}, /api/v1/jobengine/jobs/{jobId}/detail, and DAG edges /api/v1/jobengine/dag/job/{jobId}/parents and /children. These project scheduler runs into JobEngine job records; they require scheduler:read or orch:read. (Source: JobEngineJobEndpointExtensions.)
• CLI: stella orch ... covers exactly three subcommand groups — sources (list/show/test/pause/resume), backfill (start/status/list/cancel), and quotas (get/set/reset). (Source: CommandFactory.BuildOrchCommand.) There is no stella orch run start/stream subcommand — drive runs via the HTTP API above. SBOM exploration is a separate top-level command (stella sbom ..., built by CommandFactory.BuildSbomCommand), not a child of orch.

Incident response

• Queue backlog: Check queue depth with GET /api/v1/scheduler/runs/queue/lag (returns per-queue transport/queue/depth, plus totalDepth/maxDepth) or the scheduler_queue_depth gauge. Scale workers or pause sources (stella orch sources pause <id>); verify no stuck plugin.
• Repeated failures: Inspect the run record error field and the run-evidence payload from /api/v1/jobengine/jobs/{jobId}/detail (queue idempotency key, worker lease proof, quota proof, dead-letter proof, release-script proof). Compare against the originating schedule/plugin version.
• Plugin auth errors: rotate the relevant connector/source credentials in Authority/Console; the scheduler re-reads tenant-scoped credentials on the next run.
• Source/scheduler runaway: pause the offending source (stella orch sources pause); confirm the queue drains via the queue-lag endpoint or scheduler_queue_depth.

Health checks

• GET /healthz — liveness (anonymous, HTTP 200 { "status": "ok" }).
• GET /readyz — readiness (anonymous, HTTP 200 { "status": "ready" }).
• GET /api/v1/scheduler/runs/queue/lag — queue depth summary (policy scheduler:read + fine-grained scheduler.runs.read, same two-layer model as the other run reads).
• Metrics (meter StellaOps.Scheduler.Queue): scheduler_queue_enqueued_total, scheduler_queue_deduplicated_total, scheduler_queue_ack_total, scheduler_queue_retry_total, scheduler_queue_deadletter_total, scheduler_queue_depth (gauge), tagged by transport/queue.
• Metrics (meter StellaOps.Scheduler.Worker): scheduler_run_duration_seconds, scheduler_runs_active, scheduler_planner_runs_total, scheduler_runner_segments_total, scheduler_runner_backlog, scheduler_graph_jobs_total, and related counters/histograms.
• Logs/traces: the StellaOps.Scheduler.WebService activity source tags spans with tenant_id, schedule_id, run_id/job_id, http.method, and http.route (source: SchedulerTelemetryMiddleware).

Determinism/immutability

• Runs are append-only; do not mutate persisted run rows. Use new schedules/runs for fixes; retries create a fresh run linked via retryOf rather than re-running the original in place.
• Idempotency is carried on RunReason.QueueIdempotencyKey (surfaced as idempotencyKey in the JobEngine job detail). There is no runToken or inputsHash concept in the current implementation.
• State machine: Planning → Queued → Running → Completed | Error | Cancelled; terminal states are enforced by RunStateMachine (cancel/retry validate terminality).

Offline/air-gap

• Keep plugin bundles and schedule specs in sealed storage; no remote fetch.
• The graph-job backfill tool (StellaOps.Scheduler.Tools → Scheduler.Backfill) ingests GraphBuildJob payloads from a local NDJSON file (--source, --batch, --dry-run, --timeout-seconds, --pg) — useful for seeding or replaying jobs offline.
• Mutating run/schedule/source operations are written to the scheduler audit trail (ISchedulerAuditService / AddAuditEmission); export audit data through the platform audit/export tooling for offline analysis. (There is no dedicated per-run NDJSON ledger export endpoint on the scheduler service itself.)

Quick checks

• [ ] GET /healthz and GET /readyz return 200; queue depth normal (/api/v1/scheduler/runs/queue/lag).
• [ ] Latest plugin bundle signatures valid.
• [ ] No secrets in logs (spot-check redaction).
• [ ] Error budget within SLO (see docs/modules/telemetry/guides/metrics-and-slos.md).