Orchestrator Runbook (DOCS-ORCH-34-003)

Last source review: 2026-10-07

Audience: operators running the Stella Ops Orchestrator (scheduling and run execution). Purpose: pre-flight checks, day-to-day run/schedule operations, incident response, health/metrics, and offline/air-gap procedures for the Orchestrator surface.

Source verification (2026-10-07, 58fd27c7f0232bf40be341a159fe06d6931be314): Scheduler run execution is composed by src/JobEngine/StellaOps.JobEngine.WebService, registered as jobengine. Native run/schedule routes remain /api/v1/scheduler/..., with the read-only compatibility surface under /api/v1/jobengine/.... Scheduler:Worker:Embedded=true is the declared default; heavy worker registration additionally requires real JobEngine storage. Disabling the embedded flag does not make the consolidated jobengine-worker a dispatch implementation. Recheck the host Program.cs and JobEngine architecture before changing worker placement; do not restore the retired standalone Scheduler host.

Pre-flight

• Ensure PostgreSQL reachable and a queue transport (Redis or NATS) configured; liveness at GET /healthz and readiness at GET /readyz return HTTP 200 ({ "status": "ok" } / { "status": "ready" }, both anonymous). There is no aggregate /admin/health endpoint and the probes do not report queue depth — use GET /api/v1/scheduler/runs/queue/lag for depth.
• Verify tenant allowlist and scopes configured in Authority. The orchestrator authorization scopes are orch:read, orch:operate, orch:quota, and orch:backfill; the underlying scheduler surface also accepts scheduler:read, scheduler:operate, and scheduler:admin. (There is no orchestrator:* scope.) Source symbols: StellaOpsScopes.OrchRead/OrchOperate/OrchQuota/OrchBackfill, StellaOpsScopes.SchedulerRead/SchedulerOperate/SchedulerAdmin.
• Policy and handler scope checks on run/schedule endpoints. RunEndpoints and ScheduleEndpoints attach SchedulerPolicies.Read to their route groups; mutations and run preview additionally attach SchedulerPolicies.Operate. The consolidated policies require scheduler:read and scheduler:operate respectively. Handler EnsureScope checks use scheduler.runs.read/.write/.manage/.preview and scheduler.schedules.read/.write. Both production TokenScopeAuthorizer and development/test HeaderScopeAuthorizer use the same ScopeHierarchy: an exact grant satisfies the handler; scheduler:read satisfies its read checks, scheduler:operate satisfies write/manage/preview, and scheduler:admin satisfies the hierarchy. Read permission does not grant a mutation. The handler hierarchy does not replace either attached ASP.NET policy, so provision both coarse policy scopes for a normal run/schedule mutation token; additional dotted grants are not required merely to satisfy the handler. Recheck the two endpoint modules, both authorizers, ScopeHierarchy, and the policy registration in Program.cs. The JobEngine read projection endpoints use SchedulerPolicies.JobEngineRead (scheduler:read or orch:read) and no additional handler gate.
• Outside an explicit local harness, storage resolves STELLAOPS_POSTGRES_JOBENGINE_CONNECTION first, then the equally owner-specific host/test override Postgres:JobEngine:ConnectionString. The consolidated resolver does not consult predecessor Scheduler aliases or a generic shared connection. Absence fails closed. Database-less local harness mode requires TestingLocalHarness, or Scheduler:LocalHarness:Enabled=true in Development/Testing; it is not implied by a development environment alone. Recheck Program.cs and ConsolidatedJobEnginePersistenceExtensions.ResolveConnectionString.
• Plugin bundles present and signatures verified (scheduler job plugins are discovered from plugin DLLs at startup via PluginHost.LoadPlugins).

Common operations

• Create a run: POST /api/v1/scheduler/runs/ (group Read + endpoint Operate policies; handler scheduler.runs.manage). The body must reference an existing scheduleId (404 if it does not exist); only trigger: manual runs may be created through this endpoint. (Source: RunEndpoints.CreateRunAsync.)
• Cancel a run: POST /api/v1/scheduler/runs/{runId}/cancel (group Read + endpoint Operate policies; handler scheduler.runs.write). Best-effort; returns HTTP 409 if the run is already terminal. (Source: RunEndpoints.CancelRunAsync.)
• Retry a run: POST /api/v1/scheduler/runs/{runId}/retry (group Read + endpoint Operate policies; handler scheduler.runs.manage). Creates a new manual run with retryOf pointing at the original; the original must be terminal.
• Stream status: GET /api/v1/scheduler/runs/{runId}/stream — Server-Sent Events (text/event-stream), not WebSocket. Emits initial, queueLag, heartbeat, stateChanged, segmentProgress, deltaSummary, and completed events (plus a notFound event if the run row disappears mid-stream). (Source: RunStreamCoordinator.)
• List runs / read state: GET /api/v1/scheduler/runs/ (filters: scheduleId, state, createdAfter, limit, cursor, sort), GET /api/v1/scheduler/runs/{runId}, GET /api/v1/scheduler/runs/{runId}/deltas — all read endpoints apply policy scheduler:read + fine-grained scheduler.runs.read.
• Preview impact: POST /api/v1/scheduler/runs/preview (group Read + endpoint Operate policies; handler scheduler.runs.preview). Resolves the impacted image set for a scheduleId or inline selector — optionally narrowed by productKeys/vulnerabilityIds and usageOnly — and returns a bounded sample (sampleSize clamped to 1–50). Read-only; does not create a run. (Source: RunEndpoints.PreviewImpactAsync.)
• Manage schedules: under /api/v1/scheduler/schedules — list (GET /), read (GET /{scheduleId}), create (POST /), update (PATCH /{scheduleId}), soft-delete (DELETE /{scheduleId}; system-managed schedules cannot be deleted), and pause/resume (POST /{scheduleId}/pause and /resume). List/read apply policy scheduler:read + fine-grained scheduler.schedules.read; all mutations (create/update/delete/pause/resume) require both group Read and endpoint Operate policies plus handler scheduler.schedules.write. Pausing a schedule is the direct lever to stop its scheduled runs. (The CLI has no source-pause command: the former orch group was withdrawn on 2026-10-01 (SPRINT_20260928_008 PCD-129) with unsupported management calls; that withdrawal does not retire the current JobEngine reads above.) (Source: ScheduleEndpoints.)
• Console job view (read-only): GET /api/v1/jobengine/jobs, /api/v1/jobengine/jobs/{jobId}, /api/v1/jobengine/jobs/{jobId}/detail, and DAG edges /api/v1/jobengine/dag/job/{jobId}/parents and /children. These project scheduler runs into JobEngine job records; they require scheduler:read or orch:read. (Source: JobEngineJobEndpointExtensions.)
• CLI: there is no orch command group. Its sources, backfill and quotas subcommands were withdrawn on 2026-10-01 (SPRINT_20260928_008 PCD-129) with unsupported management calls. Drive runs and schedules via the HTTP API above. SBOM exploration is the separate top-level stella sbom ... command (CommandFactory.BuildSbomCommand).

Incident response

• Queue backlog: Check queue depth with GET /api/v1/scheduler/runs/queue/lag (returns per-queue transport/queue/depth, plus totalDepth/maxDepth) or the scheduler_queue_depth gauge. Check supported embedded-worker capacity or pause the offending schedule (POST /api/v1/scheduler/schedules/{scheduleId}/pause); verify no stuck plugin.
• Repeated failures: Inspect the run record error field and the run-evidence payload from /api/v1/jobengine/jobs/{jobId}/detail (available scheduler evidence; queue, lease, quota, dead-letter and release-script proof fields may be absent when no durable proof is recorded). Compare against the originating schedule/plugin version.
• Plugin auth errors: rotate the relevant connector/source credentials in Authority/Console; the scheduler re-reads tenant-scoped credentials on the next run.
• Source/scheduler runaway: pause the offending schedule (see Schedules above); confirm the queue drains via the queue-lag endpoint or scheduler_queue_depth.

Health checks

• GET /healthz — liveness (anonymous, HTTP 200 { "status": "ok" }).
• GET /readyz — readiness (anonymous, HTTP 200 { "status": "ready" }).
• GET /api/v1/scheduler/runs/queue/lag — queue depth summary (policy scheduler:read + fine-grained scheduler.runs.read, same two-layer model as the other run reads).
• Metrics (meter StellaOps.Scheduler.Queue): scheduler_queue_enqueued_total, scheduler_queue_deduplicated_total, scheduler_queue_ack_total, scheduler_queue_retry_total, scheduler_queue_deadletter_total, scheduler_queue_depth (gauge), tagged by transport/queue.
• Metrics (meter StellaOps.Scheduler.Worker): scheduler_run_duration_seconds, scheduler_runs_active, scheduler_planner_runs_total, scheduler_runner_segments_total, scheduler_runner_backlog, scheduler_graph_jobs_total, and related counters/histograms.
• Logs/traces: the StellaOps.JobEngine.Scheduling activity source tags spans with tenant_id, schedule_id, run_id/job_id, http.method, and http.route (source: SchedulerTelemetryMiddleware).

Determinism/immutability

• Use owner APIs for run transitions; do not edit persisted rows directly. The owner updates run state, progress and evidence through RunRepository.UpdateAsync / TryUpdateProgressAsync, so run rows are not an append-only ledger. Retries create a fresh run linked via retryOf rather than re-running the original in place.
• Idempotency is carried on RunReason.QueueIdempotencyKey (surfaced as idempotencyKey in the JobEngine job detail). There is no runToken or inputsHash concept in the current implementation.
• State machine: Planning → Queued → Running → Completed | Error | Cancelled; terminal states are enforced by RunStateMachine (cancel/retry validate terminality).

Offline/air-gap

• Keep plugin bundles and schedule specs in sealed storage; no remote fetch.
• The graph-job backfill tool (StellaOps.Scheduler.Tools → Scheduler.Backfill) ingests GraphBuildJob payloads from a local NDJSON file (--source, --batch, --dry-run, --timeout-seconds, --pg) — useful for seeding or replaying jobs offline.
• Mutating run/schedule/source operations are written to the scheduler audit trail (ISchedulerAuditService / AddAuditEmission); export audit data through the platform audit/export tooling for offline analysis. (There is no dedicated per-run NDJSON ledger export endpoint on the scheduler service itself.)

Quick checks

• [ ] GET /healthz and GET /readyz return 200; queue depth normal (/api/v1/scheduler/runs/queue/lag).
• [ ] Latest plugin bundle signatures valid.
• [ ] No secrets in logs (spot-check redaction).
• [ ] Error budget within SLO (see docs/modules/telemetry/guides/metrics-and-slos.md).