JobEngine Architecture

The JobEngine module is the consolidated orchestration domain for scheduled scan orchestration (Scheduler) and task-pack registry management (PacksRegistry). It is the active successor to the legacy “Source & Job Orchestrator” framing: the old StellaOps.JobEngine.* libraries (Core, Infrastructure, Worker, Tests) and the standalone TaskRunner were removed in April 2026, leaving Scheduler and PacksRegistry as the two live subdomains under src/JobEngine/.

Scope correction (2026-05-30 reconciliation): A large, generic “release control plane” — sources/quotas/throttles/incidents, circuit breakers, quota-governance, dashboards, and live deployment runs — is not part of this module. That surface is owned by the separate src/ReleaseOrchestrator/ module (gateway host release-orchestrator.stella-ops.local). Where the gateway exposes /api/v1/jobengine/* and /api/v1/release-orchestrator/* paths, most of them are translated to the Release Orchestrator service, not to JobEngine. See §7 Gateway routing for the authoritative split, and the Release Orchestrator dossier (docs/modules/release-orchestrator/) for the control-plane surface.

1) Topology

The JobEngine domain ships two deployable services. Both are .NET minimal-API microservices that register with the Stella Router and authenticate through Authority.

• Scheduler WebService (StellaOps.Scheduler.WebService). Slot 19, port 8080, gateway host scheduler.stella-ops.local. Stateless HTTP API plus, by default, the embedded worker background services (Scheduler:Worker:Embedded=true). Exposes schedule CRUD, run history, graph jobs, policy runs, policy simulations, vulnerability resolver jobs, failure signatures, workflow triggers, inbound event webhooks, and a JobEngine compatibility read surface for the Console. Authenticated via Authority scopes (scheduler:read / scheduler:operate / scheduler:admin, plus fine-grained per-action scopes — see §4 APIs).
• PacksRegistry WebService + Worker (StellaOps.PacksRegistry.*). Slot 34, port 8080, gateway host packsregistry.stella-ops.local. Stores and serves versioned task-pack bundles: pack upload/download (content, manifest, provenance), signature rotation and bulk re-envelope, DSSE attestations, lifecycle and parity state, mirror configuration + sync, offline-seed export, and a compliance summary. Authenticated via Authority scopes (packs.read / packs.write — see §4.7). There is no approval-workflow endpoint and no dedicated version-listing endpoint (GET /api/v1/packs lists packs; packs.pack_versions is a table, not a route) — earlier revisions of this dossier claimed both.
• Persistence (PostgreSQL). The Scheduler owns the scheduler schema; PacksRegistry owns the packs schema. Both auto-migrate embedded SQL on startup. There is no orchestrator schema in this module. See §5 Persistence.
• Queue abstraction. Scheduler dispatch uses a pluggable transport — Valkey/Redis Streams or NATS JetStream (StellaOps.Scheduler.Queue, Redis/ and Nats/ implementations). The selected transport is chosen via Scheduler:Queue configuration. PacksRegistry blob payloads use the seed-fs object store.
• Plugin and pack composition (updated 2026-06-06). Scheduler executable job plugins use read-only mounted bundles under /app/plugins/scheduler only when an explicit optional executable Scheduler profile or job-kind allowlist is enabled. Host-owned Scheduler jobs and declarative registry data do not require base, recommended, or harness bundle roots. The deterministic scheduler-external harness bundle is StellaOps.Scheduler.Plugin.ExternalEcho, packaged under devops/plugins/scheduler/harness/stellaops.scheduler.plugin.external-echo/ and mounted only by the harness/bad-signature overlays. PacksRegistry remains data-pack-only; its base seed root is /app/etc/plugins/packsregistry/packs and importing a pack must not execute pack-provided code.

2) Job & run model

The Scheduler operates on two related concepts:

• Schedules (scheduler.schedules): CRON-based definitions that re-evaluate already-cataloged images when intelligence changes (Concelier/Excititor/policy), driven by the BOM-Index/ImpactIndex. A schedule carries a cronExpression, timezone, mode (analysis-only by default; optional content-refresh), a Selector, optional OnlyIf/Notify/Limits, a source (user or system), and a jobKind that routes execution to a registered plugin (see §3).
• Runs (scheduler.runs): individual executions. A Run has a RunState (planning → queued → running → completed | error | cancelled), a RunTrigger (Cron, Conselier, Excitor, Manual), RunStats, deltas, an optional retryOf pointer for retries, and a RunReason JSON blob that can carry optional evidence/proof fields (see §4.5).

The lower-level scheduler.jobs queue/state table backs the vulnerability-resolver job API and generic job dispatch; per the live baseline 001_v1_scheduler_baseline.sql (line 95) it carries tenant_id, project_id, job_type, status (enum scheduler.job_status), priority, payload/payload_digest, idempotency_key, correlation_id, attempt/max_attempts, lease_id/worker_id/lease_until, not_before, reason, result, and timestamp/created_by columns. scheduler.job_history is append-only for auditability.

Correction (re-verified 2026-07-12 against 001_v1_scheduler_baseline.sql): the scheduler.jobs table does not have a task_runner_id column, and no Scheduler migration ever defined one (grep task_runner_id src/JobEngine/StellaOps.Scheduler.__Libraries/StellaOps.Scheduler.Persistence/Migrations/*.sql → no matches; note the grep must target the live baseline, not the archived pre-1.0 tree, which is excluded from embedding). The “legacy task_runner_id columns remain” claim — repeated in src/JobEngine/README.md — does not apply to the scheduler schema. If any task_runner_id legacy column exists, it would be in the Release Orchestrator module’s persistence, not here.

Lifecycle of a scheduled run:

1. Plan. The PlannerBackgroundService (and PlannerQueueDispatcherBackgroundService) evaluate due schedules and impacted images via the ImpactIndex, materializing a Run in planning/queued.
2. Dispatch. Run segments are enqueued on the configured queue transport (Valkey/NATS) with idempotency keys and retry/dead-letter handling.
3. Execute. The RunnerBackgroundService leases work and invokes the plugin registered for the schedule’s jobKind. Graph build/overlay, policy-run dispatch, and resolver workers run as additional background services.
4. Complete. Run state converges to completed/error/cancelled; RunStats/deltas are recorded and RunSummary projections updated. Operators can cancel (/runs/{id}/cancel) or retry (/runs/{id}/retry, which clones into a new run with a retryOf pointer).

Removed: The richer “enqueue → schedule → lease → complete → replay” control-plane flow with quotas/throttles/incidents and a Task Runner claim/ack bridge belongs to the Release Orchestrator module, not JobEngine. The TaskRunner service was deleted on 2026-04-08; the TaskRunner scope constants (taskrunner:read/operate/admin) were commented out in StellaOpsScopes (kept only for DB/migration backward compat). No task_runner_id column exists in the scheduler schema (see correction above).

3) Scheduler plugins

Job execution is plugin-based. ISchedulerJobPlugin (StellaOps.Scheduler.Plugin.Abstractions) exposes JobKind, ExecuteAsync(JobExecutionContext, ...), ConfigureServices(...), and MapEndpoints(...). In Program.cs the WebService builds a SchedulerPluginRegistry, registers the two host built-ins directly (Scan, then Doctor — pluginRegistry.RegisterHostCore(...), Program.cs:276,280; not the plain Register(...) overload, which is a distinct method on SchedulerPluginRegistry — the HostCore/optional-executable distinction is what makes the precedence rule below hold), then discovers additional ISchedulerJobPlugin implementations from plugin DLLs via StellaOps.Plugin.Hosting.PluginHost.LoadPlugins(...) (a duplicate JobKind from a discovered DLL is skipped, so the host built-ins take precedence on JobKind collisions).

Host built-in handlers registered in Program.cs:

• Scan (ScanJobPlugin from StellaOps.Scheduler.WebService, JobKind = "scan") — the default rescan executor.
• Doctor (SchedulerDoctorJobPlugin from StellaOps.Scheduler.WebService, JobKind = "doctor") — also maps trend endpoints under /api/v1/scheduler/doctor (see §4.6); the host calls doctorPlugin.ConfigureServices(...) at registration time.

The base runtime probe treats only those two host built-ins as required Scheduler executable surfaces. Schedule/config job kinds such as scan-image, scan-sbom, gate-evaluation, policy-simulation, notification-dispatch, partition-maintenance, and bundle-rotation remain declarative data unless a future sprint creates a signed executable job contract for them.

The standalone BundleRotationJob under the Worker library is likewise not registered as IBundleRotationScheduler, has no IAttestorBundleClient implementation, and does not consume its declared cron option. It is not evidence of a live monthly Attestor rotation path. Direct harness invocation now fails closed when tenant discovery fails; individual tenant bundle failures remain explicit result entries.

Optional executable Scheduler job plugins are admitted through signed mounted bundles. Production optional adapters now include:

• AdvisoryAI analysis (StellaOps.Scheduler.Plugin.AdvisoryAI, JobKind = "advisory-ai-analysis", profile scheduler-ai) calls the public AdvisoryAI pipeline API (POST /v1/advisory-ai/pipeline/{taskType}) and can optionally poll GET /v1/advisory-ai/outputs/{cacheKey} for cached output.
• Feed processor (StellaOps.Scheduler.Plugin.FeedProcessor, JobKind = "feed-processor", profile scheduler-feed) calls the public Concelier jobs API (POST /jobs/{*jobKind}) and can optionally poll GET /jobs/{runId} for completion.

The deterministic harness bundle is ExternalEcho (JobKind = "third-party-job", profile scheduler-external); it exercises the third-party plug-in boundary and must not be used as production AI/feed proof. audit-cleanse remains a future placeholder and must not be enabled until it has a real deletion implementation and live runtime evidence.

Additional plugin project present in the tree but not host-registered as a built-in:

• AuditCleanse (StellaOps.Scheduler.Plugin.AuditCleanse, JobKind = "audit-cleanse") — the class AuditCleanseJobPlugin exists and is unit-tested, but it is not registered in Program.cs. It is only activated if its DLL is discovered through the StellaOps.Plugin.Hosting assembly-load path.

The WebService host no longer compiles StellaOps.Scheduler.Plugin.Scan or StellaOps.Scheduler.Plugin.Doctor implementation projects directly. Optional executable job plugins remain loaded through mounted, signed assemblies. Both the scan and doctor jobs are host-owned built-ins compiled into the WebService: scan in StellaOps.Scheduler.WebService/ScanJobs/ScanJobPlugin.cs and doctor in StellaOps.Scheduler.WebService/DoctorJobs/SchedulerDoctorJobPlugin.cs (with durable trend storage in DoctorJobs/PostgresDoctorTrendRepository.cs). Their former duplicate plugin trees were removed as dead/forked code so exactly one canonical implementation of each remains: the scan dedupe dropped the unreferenced StellaOps.Scheduler.__Libraries/StellaOps.Scheduler.Plugin.Scan project and the concrete copy that had lived inside StellaOps.Scheduler.Plugin.Abstractions; the doctor dedupe dropped its duplicate projects under StellaOps.Scheduler.plugins/ and StellaOps.Scheduler.__Libraries/.

4) APIs

All routes below are served by the Scheduler WebService unless stated otherwise. Group-level authorization uses the named policies SchedulerPolicies.Read (scheduler:read), .Operate (scheduler:operate), and .Admin (scheduler:admin); most handlers additionally call EnsureScope(...) with a fine-grained scope. Tenant isolation (RequireTenant()) is enforced on all tenant-scoped groups.

4.1) Schedules — /api/v1/scheduler/schedules

Group policy: Operate is required for mutations; the group is gated on Read. Handler scopes: read = scheduler.schedules.read, write = scheduler.schedules.write.

• GET / — list schedules (filters: includeDisabled, includeDeleted, limit); includes run summaries.
• GET /{scheduleId} — schedule detail with run summary.
• POST / — create schedule (audited).
• PATCH /{scheduleId} — update schedule (audited).
• DELETE /{scheduleId} — soft-delete; system-managed schedules return 409 (audited).
• POST /{scheduleId}/pause and POST /{scheduleId}/resume (audited).

4.2) Runs — /api/v1/scheduler/runs

Group gated on Read; mutations require Operate. Handler scopes: scheduler.runs.read, scheduler.runs.write, scheduler.runs.preview, scheduler.runs.manage.

• GET / — list runs (filters: scheduleId, state, createdAfter, limit, cursor, sort); cursor pagination.
• GET /{runId} — run detail.
• GET /{runId}/deltas — run delta collection.
• GET /{runId}/stream — SSE stream of run events.
• GET /queue/lag — scheduler queue lag summary.
• POST / — create a manual run for a schedule (audited; non-manual triggers rejected).
• POST /{runId}/cancel — cancel a non-terminal run (audited).
• POST /{runId}/retry — clone a terminal run into a new run with a retryOf pointer (audited).
• POST /preview — preview impact (ImpactIndex) for a schedule/selector without scheduling.

4.3) Graph jobs — /graphs

Group policy Operate. Handler scopes use graph:read / graph:write (StellaOpsScopes.GraphRead/GraphWrite).

• POST /build — enqueue a reachability-graph build job (audited).
• POST /overlays — enqueue a graph overlay (VEX/policy) job (audited).
• GET /jobs — list graph jobs (filters by status/type).
• POST /hooks/completed — internal Cartographer callback marking a graph job complete.
• GET /overlays/lag — overlay lag metrics for SLO monitoring.

4.4) Other Scheduler endpoints

• Worker execution visibility — GET /api/v1/scheduler/workers (group Read; handler scope scheduler.workers.read). Returns exact tenant-visible/shared registration counts plus a default-20/max-100 deterministic page. The response separates embedded/standalone execution mode from registration visibility, so an empty registry does not falsely report that embedded execution is unavailable. Process ids and free-form metadata are not exposed.
• Policy runs — /api/v1/scheduler/policy/runs (group Read, mutations Operate; scope policy:run). GET /, GET /{runId}, POST /.
• Policy simulations — /api/v1/scheduler/policies/simulations (group Operate; scope policy:simulate). GET /, GET /{simulationId}, GET /{simulationId}/stream, GET /metrics, POST /, POST /preview, POST /{simulationId}/cancel, POST /{simulationId}/retry.
• Vulnerability resolver jobs — /api/v1/scheduler/vuln/resolver (group Operate; write scope effective:write, read scope findings:read). POST /jobs, GET /jobs/{jobId}, GET /metrics. Submissions persist into scheduler.jobs and survive restart.
• Failure signatures — /api/v1/scheduler/failure-signatures (group Read; scope scheduler.runs.read). GET /best-match.
• Inbound event webhooks — /events/conselier-export and /events/excitor-export (POST, anonymous group; authenticated by HMAC-SHA256 body signature and rate-limited, not by JWT/tenant headers).
• Plugin diagnostics — GET /internal/plugins/status and POST /internal/plugins/probe (Hosting/SchedulerPluginDiagnosticsEndpoints.cs:8, mapped at Program.cs:437). They enumerate registered job kinds and plugin/admission state. These routes carry no explicit scope policy; they are /internal/* and are not exposed through a gateway route, so they are reachable only in-cluster. Treat that as a network-boundary assumption, not an authorization guarantee — see the Scheduler sub-dossier (docs/modules/scheduler/architecture.md).
• Health/discovery — GET /healthz, GET /readyz (anonymous), the OpenAPI document via MapOpenApi() at /openapi/v1.json (anonymous), and a build-info endpoint (BuildInfoEndpointExtensions.MapBuildInfoEndpoint, fallback module name scheduler). The Scheduler service does not itself map /.well-known/openapi; that discovery path is a Gateway OpenApiAggregator concern, and the aggregator probes scheduler only at /openapi/v1.json (scheduler is not in the aggregator’s /.well-known/openapi switch list).

4.5) JobEngine compatibility surface

The Scheduler also exposes a read-only JobEngine compatibility surface so Console job views resolve to durable Scheduler Run state rather than placeholders. Backed by JobEngineJobEndpointExtensions; group policy SchedulerPolicies.JobEngineRead accepts either scheduler:read or orch:read.

• GET /api/v1/jobengine/jobs — list tenant-scoped compatibility records backed by Scheduler Run rows. The only compatibility jobType emitted is scheduler-run; any other requested jobType returns an empty list (no placeholders).
• GET /api/v1/jobengine/jobs/{jobId} and /{jobId}/detail — one same-tenant Scheduler-run-backed record. The detail payload is a canonical envelope (jobengine.scheduler-run.evidence.v1) containing the stored run stats, reason, deltas, retry dependency evidence, and optional proof objects (queue idempotency, worker lease, quota, dead-letter, release-script manifest, raw release-script output) only when the corresponding fields are present in scheduler.runs.reason. Absent proofs fail closed (available=false) with explicit reasons rather than fabricating data. payloadDigest is sha256: over the returned canonical payload.
• GET /api/v1/jobengine/dag/job/{jobId}/parents and /children — dependency edges derived only from durable Scheduler data; today that is scheduler.runs.retry_of retry relationships. Missing/dangling parents return no edge.

RunReason JSON can optionally carry release-script manifest fields (releaseScriptBundleId, releaseScriptBundleVersion, releaseScriptBundleDigest, releaseScriptEntryPoint, releaseScriptDependencies[]), raw output proof (release.script.raw-output.v1 — exit code, stdout/stderr byte counts and sha256: digests, and a combined digest over a stable envelope; no raw stream content), worker lease proof (scheduler.worker-lease.v1), quota proof (scheduler.quota-proof.v1, written when a run is throttled by same-tenant single-flight or max-concurrent-tenant gates), and dead-letter proof (scheduler.dead-letter.v1, written before releasing a segment that reached scheduler:queue.MaxDeliveryAttempts). All proof is stored in the existing durable scheduler.runs.reason JSONB column — no relational migration is required.

Note: The gateway also exposes /api/v1/jobengine/runs, /quotas, /deadletter, /pack-runs, /stream, /audit, /sources, and /slos, but those are translated to the Release Orchestrator service, not to this module. JobEngine itself implements only the /api/v1/jobengine/jobs and /api/v1/jobengine/dag compatibility paths. See §7.

4.6) Doctor plugin endpoints

When the Doctor plugin is loaded it maps trend endpoints under /api/v1/scheduler/doctor (gateway-routed to scheduler.stella-ops.local): GET /trends, GET /trends/checks/{checkId}, GET /trends/categories/{category}, GET /trends/degrading.

All four trend routes require the scheduler:read policy and an authenticated tenant claim. Each repository query uses that caller tenant and returns additive state="connected" and dependency="scheduler.doctor_trends" fields on success while retaining its existing collection shape (summaries, dataPoints, or checks). Missing storage, database failure, or a query that exceeds Scheduler:Doctor:TrendQueryTimeout (10 seconds by default) returns sanitized RFC 7807 503 with state="degraded", a stable reason, retryable=true, and Retry-After: 5; unavailable storage is never represented as a healthy empty trend result.

4.7) PacksRegistry endpoints

Served by the PacksRegistry WebService (packsregistry.stella-ops.local; gateway alias /api/v1/jobengine/registry/packs → /api/v1/packs). Full reference: docs/modules/packsregistry/api-reference.md.

Authorization (AUTH-1, 24da445a3f). PacksRegistry is a full Authority resource server: AddStellaOpsResourceServerAuthentication + UseIdentityEnvelopeAuthentication() → UseAuthentication() → UseAuthorization() (PacksRegistry.WebService/Program.cs:139,208-210). Every one of the 20 /api/v1 endpoints carries .RequireAuthorization(...) with one of two named policies (Security/PacksRegistryPolicies.cs), and a guard test (EveryApiEndpoint_CarriesAPacksRegistryAuthorizationPolicy) fails the build if a new endpoint is added without one:

Scopes trace to the canonical catalog (StellaOpsScopes.PacksRead / PacksWrite = packs.read / packs.write, StellaOpsScopes.cs:434,439) and are seeded as assignable permissions (S001_v1_authority_operational_baseline.sql:444,446). No service-local scope strings exist.

An optional shared X-API-Key (PacksRegistry:Auth:ApiKey) is a supplementary gate layered on top of scope authorization — when it is unset the API-key check is skipped but scope authorization still applies (Program.cs:1296-1312); the comparison is constant-time (FixedTimeEquals, :1305,1314). It is not an authentication mechanism and never was a substitute for scopes.

Historical note. Before AUTH-1 this service enforced no scope authorization at all: the only gate was that optional API key, which returned “authorized” when unset. Any principal the gateway admitted into a tenant could rotate pack signatures, bulk re-envelope packs (an endpoint that accepts an operator private key in the request body), reconfigure mirrors, and export a full-tenant offline seed. Fixed 2026-07-12.

Tenancy. The data-isolation tenant is resolved exclusively from the validated stellaops:tenant claim via UseStellaOpsTenantMiddleware + TryRequireTenant (Program.cs:212,1321+); a tenant supplied in a query string or body must equal the claim or the request is rejected. Caller input is never the isolation key.

Endpoints (all /api/v1, W = packsregistry.write, R = packsregistry.read):

/healthz and the static OpenAPI stubs (/openapi/packs.json, /openapi/pack-manifest.json) are anonymous by design.

There is no approval endpoint — lifecycle state (POST /packs/{packId}/lifecycle) is the nearest surface but is not an approval workflow — and no dedicated version-listing endpoint.

5) Persistence (PostgreSQL)

Both subdomains auto-migrate embedded SQL on startup; SQL migrations are authoritative and EF Core contexts are scaffolded from them.

Scheduler — schema scheduler(SchedulerDbContext; default schema name hard-coded as SchedulerDataSource.DefaultSchemaName = "scheduler", used by every Postgres repository’s GetSchemaName()). The live migration set is exactly two files — 001_v1_scheduler_baseline.sql (the collapsed pre-1.0 chain) plus the seed baseline S001_v1_scheduler_seed_baseline.sql (seed is opt-in via SCHEDULER_BOOTSTRAP_ENABLED=true). The old per-feature files (001_initial_schema.sql, 002_graph_jobs.sql, 003_runs_policy.sql, 010–012b, and the mig061 set 001–009 + S001_demo_seed.sql) live under Migrations/_archived/pre_1.0/ and are excluded from embedding — grep the baseline, not the archived tree. Key tables (baseline 001_v1_scheduler_baseline.sql, scheduler.jobs at line 95): jobs, triggers, workers, locks, job_history, metrics, schedules, runs, impact_snapshots, run_summaries, execution_logs, graph_jobs, graph_job_events, policy_jobs, policy_run_jobs. Enum types include job_status, graph_job_type, graph_job_status, run_state, policy_run_status. The deprecated local scheduler.audit partitioned table (and all partitions) is created and then dropped inside the same baseline (DROP TABLE IF EXISTS scheduler.audit CASCADE;, line 1152); audit now flows to the Timeline unified audit sink (timeline.unified_audit_events). The baseline also folds in the scripts schema, schedule source/schema_version/job_kind columns, an HLC queue chain, exception lifecycle, and Doctor trends.

Live Scheduler hosts resolve graph jobs, schedules, runs, run summaries, policy-run state, audit, and resolver-job state through PostgreSQL-backed repositories whenever a storage connection string is configured (Scheduler:Storage:ConnectionString, Scheduler:Storage:Postgres:Scheduler:ConnectionString, or Postgres:Scheduler:ConnectionString). They fail fast on startup when none is present, outside explicit local-harness mode. In-memory repositories are reachable only via the TestingLocalHarness environment or Scheduler:LocalHarness:Enabled=true in Development/Testing.

PacksRegistry — schema packs(PacksRegistryDataSource; one live migration, 001_v1_packsregistry_baseline.sql — the pre-1.0 001_initial_schema.sql and 002_runtime_pack_repository_alignment.sql were folded into it and now sit under Migrations/_archived/pre_1.0/mig061/, excluded from embedding). Storage:Driver=postgres is the production default for metadata/state repositories; blob payloads use the seed-fs object store (SeedFsPacksRegistryBlobStore). The live host accepts only postgres and filesystem drivers (the old inmemory branch was removed). Startup fails fast when Storage:ObjectStore:Driver is rustfs (not implemented) or any unsupported value, and when Storage:Driver=postgres is set without a connection string outside Development.

There is no OrchestratorDbContext, JobEngineDbContext, orchestrator schema, or 011_compatibility_deployments.sql migration in this module. Tables such as sources, quotas, throttles, incidents, and compatibility_deployments are owned by the Release Orchestrator module.

6) Background workers & runtime modes

By default (Scheduler:Worker:Embedded=true) the worker background services run in the WebService process, so a separate scheduler-worker container is not required; set Embedded=false and deploy StellaOps.Scheduler.Worker.Host to scale workers independently. Registered hosted services include PlannerBackgroundService, PlannerQueueDispatcherBackgroundService, RunnerBackgroundService, PolicyRunDispatchBackgroundService, GraphBuildBackgroundService, and GraphOverlayBackgroundService, plus exception/notification workers (ExceptionLifecycleWorker, ExpiringNotificationWorker) and SystemScheduleBootstrap, which run in the web process regardless of embedded mode. Additional workers exist for resolver evaluation, reachability, policy re-evaluation/reconciliation, and simulation reduction.

Runtime guards:

• Authority authentication is mandatory outside Testing or explicit local-harness mode; Scheduler:Authority:Enabled=false / AllowAnonymousFallback=true are rejected before startup in production-like environments. Header-based X-Tenant-Id/X-Scopes auth is available only in Testing, TestingLocalHarness, or Development with Scheduler:LocalHarness:Enabled=true.
• Scheduler’s by-CVE ImpactIndex path uses Scheduler:Worker:Concelier to query Concelier GET /advisories/observations?alias=... for CVE/GHSA alias -> linkset.purls projection. Concelier resolves tenancy from the validated stellaops:tenant claim, not from X-StellaOps-TenantId, so production compose enables Scheduler:Worker:Concelier:Authority and mints a tenant-scoped client-credentials bearer with the dedicated stellaops-scheduler-concelier-observations service account. That account is limited to the vuln:view scope and stellaops audience; the resolver adds the run tenant as the token tenant parameter.
• Inbound Conselier/Excitor webhook rate limiting uses a Redis-backed distributed sliding-window limiter from the scheduler:queue transport contract; the process-local limiter is available only in TestingLocalHarness. Startup fails fast if inbound webhooks stay enabled without Redis queue configuration. When both inbound webhooks are explicitly disabled, the disabled limiter throws if invoked rather than silently allowing traffic.
• Worker event publishing, exception-lifecycle notifications, and expiring-exception alerts publish through the Notify event queue when notify:queue is configured; missing wiring fails closed outside explicit harnesses (Scheduler:Worker:Notifications:UseLocalHarness=true enables the local publisher in Development).
• Disabled Cartographer webhooks resolve a disabled client that makes no outbound call; graph-job completion events still publish through the configured graph-job event publisher.

Primary configuration sections: Scheduler:Authority, Scheduler:Worker (incl. :Embedded, :Notifications:UseLocalHarness, :Concelier), Scheduler:Events (inbound webhooks), Scheduler:Cartographer, Scheduler:Queue (incl. :Hlc:DsseSigning), Scheduler:RunStream, Scheduler:Storage, Scheduler:LocalHarness:Enabled, and Router.

7) Gateway routing

The router (src/Router/StellaOps.Gateway.WebService/appsettings.json) splits the jobengine namespace across services. Only the rows targeting scheduler.stella-ops.local / packsregistry.stella-ops.local belong to this module:

There is no orchestrator.stella-ops.local host; the legacy /api/orchestrator path is translated to the Release Orchestrator service.

8) Observability

Scheduler metrics use a scheduler_* / graph_* / policy_simulation_* naming convention (see StellaOps.Scheduler.Queue/Metrics and the worker observability classes), for example: scheduler_queue_depth, scheduler_queue_enqueued_total, scheduler_queue_deduplicated_total, scheduler_queue_ack_total, scheduler_queue_retry_total, scheduler_queue_deadletter_total, scheduler_runner_backlog, scheduler_hlc_enqueues_total / _duplicates_total / _enqueue_latency_ms, scheduler_chain_verifications_total / _failures_total, scheduler_batch_snapshots_total, graph_jobs_inflight, graph_build_seconds, overlay_lag_seconds, policy_simulation_queue_depth, and policy_simulation_latency.

Structured logs and audit events carry tenant/run/schedule context and flow to the Timeline unified audit sink. Run progress is streamed over SSE (/runs/{id}/stream, /policies/simulations/{id}/stream).

The generic job metrics named job_queue_depth / job_latency_seconds / job_failures_total / job_retry_total / lease_extensions_total, and the pack_run_* Task Runner metrics, are not emitted by this module (Task Runner was removed; the generic-orchestrator metrics belong to the Release Orchestrator module).

9) Sprint 208 consolidation (orchestration domain subdomains)

Sprint 208 consolidated the Scheduler, TaskRunner, and PacksRegistry source trees under src/JobEngine/ as subdomains of the orchestration domain. Each subdomain retains its own project names, namespaces, and runtime identities; no namespace renames were performed. TaskRunner was subsequently removed (2026-04-08), and the legacy StellaOps.JobEngine.Core/.Infrastructure/.Worker/.Tests libraries were removed in April 2026 (no active service depended on them; Release Orchestrator now uses its own StellaOps.ReleaseOrchestrator.Persistence). The TaskRunner scope constants are commented out in StellaOpsScopes (kept only for DB/migration backward compat). The scheduler schema has no task_runner_id column (see §2 correction).

9.1) Scheduler subdomain

Source: src/JobEngine/StellaOps.Scheduler.*. Re-evaluates already-cataloged images when intelligence changes, orchestrates nightly and ad-hoc runs, targets only impacted images via the ImpactIndex, and emits report-ready events for Notify. Default mode is analysis-only (no image pull); content-refresh is opt-in per schedule. Deployables: StellaOps.Scheduler.WebService (with embedded worker by default) and the optional standalone StellaOps.Scheduler.Worker.Host. Database: SchedulerDbContext, schema scheduler (see §5).

9.2) TaskRunner subdomain (REMOVED)

TaskRunner was deleted on 2026-04-08. Source directories, Docker services, CLI commands, and docs were removed. The TaskRunner scope constants (taskrunner:read/operate/admin) were commented out in StellaOpsScopes. No new migrations were created for the removal, and no task_runner_id column exists in the scheduler schema (the “legacy columns remain” claim was never true here).

9.3) PacksRegistry subdomain

Source: src/JobEngine/StellaOps.PacksRegistry/ and StellaOps.PacksRegistry.__Libraries/. Manages compliance/automation pack definitions, versions, and distribution. Deployables: StellaOps.PacksRegistry.WebService, StellaOps.PacksRegistry.Worker. Storage contract per §5: Postgres metadata/state (schema packs) plus seed-fs object-store payload channels; postgres/filesystem drivers only.

10) ADR: No DB merge (Sprint 208)

Decision: the orchestration subdomains keep separate DbContexts and separate PostgreSQL schemas; no cross-schema DB merge. Sprint 208 evaluated merging the then-existing Orchestrator DbContext and the Scheduler DbContext (both defined Jobs/JobHistory entities with incompatible semantics — pipeline orchestration runs vs. cron-scheduled rescan executions). Merging would have required renaming one entity set and regenerating compiled models, for no operational benefit since the schemas already provide clean separation in the same stellaops_platform database.

Current consequence: the Orchestrator DbContext referenced by this ADR is no longer part of the JobEngine module — the standalone orchestrator persistence moved to the Release Orchestrator module and the legacy JobEngine libraries were removed. Within JobEngine, the surviving contexts are SchedulerDbContext (schema scheduler) and the PacksRegistry repositories (schema packs), which remain independent.

11) Schema continuity note (Sprint 221 / 311 — historical)

Sprint 221 renamed the orchestration domain from “Orchestrator” to “JobEngine” but preserved the PostgreSQL schema name orchestrator for the then-existing orchestrator persistence; Sprint 311 aligned runtime, design-time, and compiled-model paths on that preserved default (JobEngineDbContext.DefaultSchemaName = "orchestrator"). That persistence layer no longer lives in this module — it was removed with the legacy JobEngine libraries, and any surviving orchestrator-schema persistence is owned by the Release Orchestrator module. The active JobEngine contexts use schemas scheduler and packs; no physical schema rename was ever applied, and any future rename remains a dedicated migration-sprint concern for whichever module owns the schema.

12) Worker SDKs under src/JobEngine/ (Go, Python) — orphaned, not a live contract

Two cross-language client SDKs ship as source under this module:

• src/JobEngine/StellaOps.JobEngine.WorkerSdk.Go/ — Go module git.stella-ops.org/stellaops/jobengine/worker-sdk-go, package pkg/workersdk.
• src/JobEngine/StellaOps.JobEngine.WorkerSdk.Python/ — package stellaops_jobengine_worker (+ sample_worker.py).

They implement a worker control-plane protocol that no service in this repository serves. Both clients speak claim/ack/heartbeat/progress against these routes (WorkerSdk.Go/pkg/workersdk/client.go:88,101,131,163; WorkerSdk.Python/stellaops_jobengine_worker/client.py:50,70,79,92):

A repo-wide search for /api/jobs/ in src/** returns only the SDKs themselves and their own self-mocking tests (client_test.go:33,51,53, test_client.py:46,58,60, which stand up an in-process fake HTTP server). There is no /api/jobs/* route in JobEngine, in Release Orchestrator, or anywhere else — and no gateway route translates to one. The SDK unit tests pass against their own fake, so green CI proves nothing about a real server.

This is the removed TaskRunner/legacy-Orchestrator contract. The banner at the top of this dossier records that the old StellaOps.JobEngine.* libraries (Core, Infrastructure, Worker, Tests) and the standalone TaskRunner service were deleted in April 2026. The SDKs are the client half of that deleted server and were never removed with it. The Python README still calls itself the “StellaOps Orchestrator Worker SDK” and reads ORCH_BASE_URL / ORCH_API_KEY.

Attribution correction — they are not Release Orchestrator’s contract either. It would be reasonable to assume (as the 2026-07-11 audit did) that these SDKs simply target Release Orchestrator and are misfiled. They do not. RO’s external-worker surface is agent-scoped and shaped entirely differently (ReleaseOrchestrator.WebApi/Endpoints/AgentRuntimeEndpoints.cs:24-33):

• POST /api/v1/release-orchestrator/agent-runtime/agents/{agentId:guid}/tasks:poll
• POST .../agents/{agentId:guid}/tasks/{taskId:guid}/result
• POST .../agents/{agentId:guid}/heartbeat

— authenticated by mTLS client certificate matching a registered agent’s thumbprint. The SDKs authenticate with Authorization: Bearer <api-key> plus raw X-StellaOps-TenantId / X-StellaOps-Project headers (client.go:208-219) — i.e. the forgeable-header tenancy model the platform has since abandoned in favour of the claim-bound stellaops:tenant contract. There is no lease/claim concept in RO’s agent protocol at all. So “relocate the SDKs next to the module that serves claim/heartbeat/progress” is not actionable: no module serves it.

Status: orphaned / non-functional. Do not build against them. They are not packaged, published, or referenced by any Dockerfile or compose file (grep -rn WorkerSdk devops/ → no matches). The Go SDK is still exercised by a report-only CI lane (.gitea/workflows/go-tests.yml, matrix entry jobengine-worker-sdk) against its own mock. Their AGENTS.md charters point integrators at this dossier as required reading, which is why this section exists: if you arrived here from those charters, the worker protocol they describe does not exist. Scheduler dispatches work to in-process plugins (§3), not to external claim/heartbeat workers.

Disposition (open). Removal is the likely correct end state but is deliberately not taken here: it is a dead-code decision with a CI/manifest blast radius (.gitea/workflows/go-tests.yml, docs/technical/testing/TEST_MANIFEST.yml:157,176-177,198, docs/technical/testing/test-suite-layer-inventory.csv, test-project-ci-inventory.csv) and belongs in a scoped dead-code sprint alongside the other survivors of that gate. Until then this section is the honest record. Tracked in SPRINT_20260712_009 (CLO-5).

Related documents

• docs/modules/jobengine/README.md — module overview.
• docs/modules/jobengine/event-envelope.md — notifier/webhook/SSE envelope draft for job/run events.
• docs/modules/jobengine/job-export-contract.md — deterministic job/run export payload for Findings Ledger.
• Release Orchestrator dossier (docs/modules/release-orchestrator/) — the control-plane surface (sources, quotas, throttles, incidents, circuit breakers, deployments) that the gateway exposes under several /api/v1/jobengine/* and /api/v1/release-orchestrator/* paths.