JobEngine Architecture

Verified against commit 942a18727c (2026-09-26). Re-verify with: rg -n "Map(Get|Post|Patch|Delete|Group)\(" src/JobEngine/StellaOps.JobEngine.WebService src/JobEngine/StellaOps.JobEngine.__Libraries/StellaOps.JobEngine.Scheduling && ls src/JobEngine/StellaOps.JobEngine.__Libraries/StellaOps.JobEngine.Persistence/Migrations && rg -n "^ jobengine-(web|worker|estate-worker):" devops/compose devops/release/bundle/docker-compose.yml

JobEngine is the consolidated family for scheduled scan orchestration (the Scheduler subdomain) and Task Pack registry management (the PacksRegistry subdomain). One web host serves both surfaces, and the family owns one database.

The generic release control plane (sources, quotas, throttles, incidents, dead-letter views, deployment runs, SLO read models) is not part of this module. It belongs to src/ReleaseOrchestrator/ (see Release Orchestrator).

1) Topology

Libraries:

• StellaOps.JobEngine.__Libraries/StellaOps.JobEngine.Scheduling/ — the Scheduler application layer (endpoints, options, host-owned job plugins, authorization helpers).
• StellaOps.JobEngine.__Libraries/StellaOps.JobEngine.Persistence/ — the family’s only data access layer and migration authority. The Scheduler DAL (Scheduling/) and the Task Pack DAL (Packs/) keep their original StellaOps.Scheduler.Persistence.* and StellaOps.PacksRegistry.Persistence.* namespaces; SchedulerDbContext lives here.
• StellaOps.JobEngine.__Libraries/StellaOps.JobEngine.Approvals/ — the estate-pack run-approval gate.
• StellaOps.Scheduler.__Libraries/ — Models, Queue, Worker, ImpactIndex, DoctorOperations, Plugin.Abstractions and the optional Plugin.* job plugins. These keep their names; three of them ship inside the signed estate-doctor producer bundle.
• StellaOps.PacksRegistry/ (.Core, .Infrastructure) and StellaOps.PacksRegistry.__Libraries/StellaOps.PacksRegistry.OfflineSeed.Contracts/ — the Task Pack domain and the offline-seed verifier SDK.
• StellaOps.Scheduler.Tools/Scheduler.Backfill/ — a backfill tool.

Retired and frozen under src/__Obsoleted/JobEngine/ (CoC §15.4; do not edit or reference): StellaOps.Scheduler.WebService, StellaOps.Scheduler.Worker.Host, StellaOps.PacksRegistry.WebService, StellaOps.PacksRegistry.Worker, and the predecessor StellaOps.Scheduler.Persistence / StellaOps.PacksRegistry.Persistence projects (kept only for their applied migration history). The scheduler-web, packsregistry-web and packsregistry-worker deployables were removed at JOB-9 (2026-09-12); the scheduler and packsregistry hostname aliases were removed at JOB-10 (2026-09-13). The live gateway host is jobengine.stella-ops.local.

2) Persistence

Tenant run-usage measurement

GET /api/v1/jobengine/quotas/usage returns the authenticated tenant’s count of persisted scheduler.runs records created in the current UTC day. The interval includes its start and excludes the next day’s start. Every state and retry record counts; list pagination does not cap this aggregate. The owner supplies the tenant, unit, period bounds and measurement time under the jobengine.run-quota.v1 contract. It supplies usage rather than licence entitlements, operator ceilings or promotion decisions.

The read requires quota.read or orch:quota and tenant binding. Quota permission does not grant the Scheduler run-list read. Platform consumes this owner API through STELLAOPS_JOBENGINE_URL; it never connects to the JobEngine database.

Tenant-boundary review (PCD-425: serving source 4d2839563c, acceptance implementation e7b27c4e2b, 2026-10-06 UTC). The JWT host registers ClaimsTenantContextAccessor; authentication and authorization run before tenant middleware. The quota handler reads the authenticated stellaops:tenant claim through that accessor and passes its trimmed tenant to DailyRunQuotaUsageService, which forwards it to the owner usage reader. For ordinary callers, a tenant header cannot replace the signed claim. A missing or blank signed tenant returns HTTP 400 tenant_missing before the usage reader runs, including when a caller supplies a spoofed canonical or harness tenant header. Scope failures and anonymous requests return HTTP 403 and 401 without usage reads.

The HTTP acceptance tests boot TestingLocalHarness, use the host’s own tenant/scope registrations and observe the actual harness reader without replacing its counts. Separate PostgreSQL reader tests verify persisted tenant/day isolation, reader recreation and unattributed-input refusal. These are source and isolated runtime-fixture checks; they do not attest the deployed estate.

Re-verify with RunQuotaUsageReaderTests in the Scheduler persistence project and RunQuotaUsageEndpointTests in its web-service test project, using tools/scripts/test-targeted-xunit.ps1 -Class "*RunQuotaUsage*" -BuildProjectReferences.

• Database. stellaops_jobengine, owned by the jobengine role on the shared PostgreSQL server. Every role resolves STELLAOPS_POSTGRES_JOBENGINE_CONNECTION (or the equally scoped configuration key Postgres:JobEngine:ConnectionString) through ConsolidatedJobEnginePersistenceExtensions.ResolveConnectionString and fails closed without it. There is no shared-connection fallback. The web host allows a database-less start only in explicit local-harness mode (TestingLocalHarness, or Scheduler:LocalHarness:Enabled=true in Development/Testing).

• Schemas. scheduler, scheduler_app (tenant-context helper functions used by row-level security), packs, and eventing (the shared eventing reliability chain, including the fenced lease in eventing.leases).

• Owner datasource. The consolidated registration provides one bare NpgsqlDataSource in every family role, using the same JobEngine connection resolver and connection policy as startup migrations. Retention pruners and Doctor resolve that owner datasource; Web does not register a second pool. Registering the datasource permits retention to run; the existing age, publication, checkpoint and declared-consumer eligibility rules still decide which rows can be reclaimed. Verified against 22f9674b151f1909f2810faf026a679cd4904e16 on 2026-10-06; re-verify with the ConsolidatedJobEngineSchemaTests.RetentionHostComposition cases and central eventing retention tests. PCD-402 evidence (pruned evidence docs/implplan/_evidence/20261006-scanner-jobengine-carry/PCD-402.md at cf69103887) records the two worker image IDs and live startup proof.

• Migrations. Embedded SQL under src/JobEngine/StellaOps.JobEngine.__Libraries/StellaOps.JobEngine.Persistence/Migrations/ (single-level glob; archived lineages are never embedded). Web, worker and estate worker all converge the database on startup; the runner serializes them on an advisory lock. Applied files are immutable (forward-only). The chain:

  Migration	Content
  001_jobengine_consolidated_baseline.sql	scheduler, scheduler_app and packs baseline
  002–005	Estate schedule constraints and dispatch ledger, run approvals and approval audit, run dispatches, run queue
  006_dual_mode_pack_storage.sql	Dual-mode pack storage: seed generations, signature floors/history, storage events/outbox, upload capacity reservations
  007_estate_pack_identity_canonical_backfill.sql	Re-keys estate packs to <manifest.packId>@<manifest.version>
  008_offline_seed_import_anchors.sql, 009_packs_import_anchors_retention_class.sql	Deferred transparency-log anchors for admitted offline-seed imports
  010_retire_unused_scheduler_tables.sql	Drops the empty doctor_trends, execution_logs, graph_job_events and run_summaries tables; refuses a populated table
  011_jobengine_terminal_retention.sql, 012_jobengine_retention_classes.sql	Terminal-state retention routines and per-table retention classes
  013_scanner_image_projection.sql, 014_advisory_trigger_pending.sql	Tenant Scanner-feed projection and pending advisory product keys
  015_capacity_probe_receipt.sql	Owner-wide capacity receipts in the existing scheduler schema; newest 90 retained

• No demo data. No migration inserts sample schedules, jobs, scripts or packs. SCHEDULER_BOOTSTRAP_ENABLED (default false in compose) gates only bootstrap seed migrations.

• Row-level security. The owner role is not a superuser and the tenant tables force RLS. The cross-tenant worker loops therefore run on a system connection that carries the trigger loop’s own tenant list (Scheduler:Worker:Trigger:Tenants) when exactly one tenant is configured.

• Impact snapshot reads. The runner reads an impact snapshot with its persisted run’s tenant, through a tenant-pinned reader connection and an explicit snapshot-id/tenant-id SQL predicate (PCD-353). Another tenant’s snapshot returns no result; the read does not rely on the single-tenant system-connection fallback. The owning non-superuser role’s FORCE RLS is exercised independently of that predicate in the PostgreSQL contract fixture. Verified against a2527f11e3; reverify with pwsh tools/scripts/test-targeted-xunit.ps1 -Project src/JobEngine/StellaOps.JobEngine.__Tests/StellaOps.JobEngine.Persistence.Tests/StellaOps.JobEngine.Persistence.Tests.csproj -Class '*TenantScopedDataSourcesTests*' -BuildProjectReferences -Restore.

• Not in this database. The scripts schema and release-control-plane storage belong to Release Orchestrator. The former stellaops_platform.scheduler, scheduler_app and packs schemas were dropped on 2026-09-14 (PLT-4).

• Retention. Terminal-state cleanup, row budgets, protected records and the retention Doctor check are described in Retention and custody.

3) Job and run model

• Schedules (scheduler.schedules): cron definitions that re-evaluate already-cataloged images when intelligence changes. A schedule carries cronExpression, timezone, mode (analysis-only by default; content refresh is opt-in), a selector, optional onlyIf/notify/ limits, a source (user or system) and a jobKind that selects the executing plugin.
• Triggers. ScheduleTriggerService polls due schedules (Scheduler:Worker:Trigger:Enabled, :PollInterval, :MisfireGrace, :Tenants). The advisory trigger (below) creates runs when a Vulnerabilities hub fact change concerns indexed images.
• Runs (scheduler.runs): a run moves planning → queued → running → completed | error | cancelled, carries a trigger (Cron, Manual, Advisory; Conselier and Excitor remain only so runs stored by the retired export webhooks still read), stats, deltas, an optional retryOf pointer and a reason JSONB document. Optional proof objects (release-script manifest, raw-output digests, worker lease, quota, dead-letter) are stored in reason; they back the compatibility detail view in §5.2.
• Runner progress. A scan segment captures its start time before Scanner calls and its finish time after them. It writes the merged stats through IRunRepository.TryUpdateProgressAsync, comparing the persisted completed count as a monotonic version and allowing only an active queued/running run. A concurrent writer re-reads and merges its own image results into the latest stats, for at most eight update attempts. Exhaustion throws so the segment remains retryable. The update preserves retryOf and schemaVersion; a completed run’s duration includes the last segment’s Scanner work.
• Schedule summaries. The durable service reconstructs counters from scheduler.runs on every read. It keeps no projection cache or separate high-water map. For each non-retired schedule it selects the latest retained run by creation time, then .NET ordinal run ID, within the exact tenant boundary. Disabled schedules remain eligible. One lateral indexed lookup per schedule reads the latest timestamp’s candidates; timestamp ties use StringComparer.Ordinal so valid Unicode IDs keep the same order as PCD-401. Counters describe one selected run, not historical totals. A host restart needs no replay or warm-up. Producers persist before calling ProjectAsync; that call reads current retained state and never writes a projection. Delayed terminal, active, progress, older-run or deleted-run snapshots cannot replace counters. The persisted completed-count guard described above remains the progress fence. Retirement removes summary eligibility immediately, even while run rows remain. Run deletion selects the latest remaining retained run; when none remains, GetAsync returns null and ListAsync omits that schedule. A projection call without an eligible retained run returns zero counters and null LastRunId. No cache or tombstone can accumulate after retirement. The separate lastRunAt read also uses retained runs. A missing summary or timestamp does not establish that a schedule has never executed. Database failures propagate rather than serving stale cached counters. This contract applies to durable storage; the explicitly enabled local harness remains volatile. Re-verify with RunSummaryLifecycleHostTests, RunSummaryServiceOrderingTests, RunRepositoryTests and the adjacent schedule/run host specs through the targeted xUnit wrapper. Verified against 52f73ca66085102eb606d5a586890b68a30d648f on 2026-10-06. The PCD-405 receipt (pruned evidence docs/implplan/_evidence/20261006-jobengine-summary-lifecycle/PCD-405.md at f8dbcae9e3) records the source provenance and red/green controls.
• Abandoned runs. The runner reconciles the oldest 50 queued and 50 running rows between batches, once per minute by default. A run older than Scheduler:Worker:Runner:Execution:MaximumRunAge (default 24 hours from creation) closes as Error with run_maximum_age_exceeded, preserving its stats and original start. The completed-count version check prevents overwriting concurrent progress or terminal states. Fresh runs remain active. On an empty queue poll, pending entries idle for that budget are reclaimed; entries for terminal runs are acknowledged without requesting Scanner reports. This bounds abandoned work without stealing leases inside the execution budget. RunReconciliationInterval must be positive; the budget must be at least the lease duration and at most 30 days. This rule was verified against 98ffa2a68f plus PCD-306 on 2026-10-05; recheck RunnerBackgroundServiceProofTests and RunnerBackgroundService.CloseOverdueRunsAsync.
• Jobs (scheduler.jobs, scheduler.job_history): the lower-level queue/state table behind the vulnerability-resolver job API; job_history is append-only.

Lifecycle: the planner (PlannerBackgroundService, PlannerQueueDispatcherBackgroundService) evaluates schedules against the ImpactIndex and materializes runs; segments are enqueued on the configured queue transport with idempotency keys and dead-letter handling; RunnerBackgroundService leases scan segments and requests Scanner reports; graph build/overlay and policy-run dispatch run as further hosted services. Operators cancel with POST /api/v1/scheduler/runs/{runId}/cancel and retry with .../retry (a clone with retryOf).

Retry contract (CCP-032). POST /api/v1/scheduler/runs/{runId}/retry requires the Operate policy and the scheduler.runs.manage scope. It accepts a terminal run (completed, error or cancelled) that belongs to an existing schedule; otherwise it answers 404 (unknown run), 400 (no schedule, or the schedule no longer exists) or 409 (not terminal) and stores nothing. A source run is retried at most once. The retry run’s id is derived from the tenant and the source run id (run_ plus the first 32 hex characters of SHA-256 over stellaops.scheduler.run-retry.v1, the tenant and the source run id, newline-separated), and the insert stores nothing when that id exists. The first request returns 201 with the new run, writes one scheduler.run retry audit event naming the actor, and projects the run summary. A repeated request, or a concurrent one that loses the insert, returns 200 with the recorded retry and adds no run, projection or audit event; a retry recorded before this contract (with a generated id) is found through retry_of and returned the same way. A further attempt is a retry of the retry run. The outcome is attributable on the owner record: the retry run carries retryOf, and GET /api/v1/jobengine/dag/job/{runId}/children lists it as a retry edge. There is no dead-letter API and no dead-letter retry: the queue transports keep their own dead-letter handling. A run’s dead-letter proof (Reason.DeadLetterProof) is read by the job detail evidence and by SchedulerOperationsHealthProbe.GetDeadLetterStatusAsync, the source of the Doctor check check.operations.dead-letter; no current host registers that probe.

The ImpactIndex (IImpactIndex, implemented by RoaringImpactIndex) holds one record per tenant and image digest: registry, repository, namespaces, tags, labels, and the image’s components as package URLs, each with a usage of Entrypoint, NotUsed or Unknown. IngestAsync adds or replaces one tenant’s record and RemoveAsync(tenantId, digest) deletes it; the same digest in two tenants is two records. Selector planning reads it with ResolveAllAsync, ResolveByPurlsAsync and ResolveByVulnerabilitiesAsync. Each component purl is also filed under its Vulnerabilities hub product key (a versionless purl, ProductKeyV1; apk and deb packages under their source package), so a by-purl query for pkg:npm/lodash matches every image carrying any lodash version. SelectIndexedAsync returns which of a set of purls or product keys match an image a selector admits. With usageOnly (analysis-only schedules), a by-purl match is dropped only when that component is known not to be used, and an image is dropped from ResolveAllAsync only when every one of its components is known not to be used.

The index is filled from Scanner’s retained scanner.scan.completed v2 feed by the Scanner image projection (Scheduler:Worker:ScannerProjection): the application remains opt-in, while the release bundle and dev stack default JOBENGINE_SCANNER_PROJECTION_ENABLED to true after the PCD-354 cadence ruling. An explicit false override remains supported, and missing required credentials fail startup closed.

• Feed. For each tenant in Scheduler:Worker:Trigger:Tenants (default default) a hosted worker reads GET /api/v1/scans/completed/events from its durable checkpoint, under the per-tenant fenced lease jobengine:scanner-image-projection:tenant:*, and reports its position to Scanner as consumer jobengine.scheduler-impact-v1. Each event is applied in one transaction with its inbox admission and checkpoint advance. A gap, or a retention gap behind an existing checkpoint, stops that tenant fail-closed; a new producer epoch retained from sequence one resets the tenant’s rows and replays. A tenant’s FIRST consumption of a stream whose oldest events Scanner has already pruned starts at the retention horizon (logged as a warning): an image scanned only before the horizon enters the index when it is scanned again.
• Persistence. scheduler.scanner_image_projection (migration 013, row-level security on tenant_id, class derived-rebuildable) keeps one row per tenant and image: location, tags, component package URLs, last completed scan. The index is rebuilt from it before the planner starts, so a restarted host plans from the database without reading the feed again.
• Location. The event’s image reference gives the registry, repository and tag. An event without one keeps the stored location and replaces only the components; a digest first seen without a reference is stored under registry unresolved with the digest as its repository, so AllImages schedules plan it and ByRepository selectors do not.
• Usage. The event carries no per-component usage, so every component is Unknown, which usage-only planning keeps.
• Age window. An image whose last completed scan is older than RetentionDays (default 30) leaves the table and the index; scheduled rescans refresh it, so in practice an image ages out when its rescans stop succeeding.
• Identity. The dedicated client stellaops-jobengine-scanner-projection holds only scanner:projection:read (secret JOBENGINE_SCANNER_PROJECTION_CLIENT_SECRET), one token per exact tenant, in its own token-client container.
• Replicas. Only the lease holder applies events, and only its in-memory index updates at once; jobengine-web runs one replica, and another replica would see new images only after a restart.

The advisory trigger (Scheduler:Worker:VulnerabilityHub:Trigger, default off; it requires the Scanner image projection) replaced the retired Concelier and Excititor export webhooks:

• Feed. For each scheduler tenant a hosted worker reads the hub’s global vuln.facts stream (GET /api/vulnerabilities/v1/events/vuln.facts) from its durable checkpoint, under the per-tenant fenced lease jobengine:advisory-trigger:tenant:*. vuln.fact.upserted and vuln.fact.tombstoned name a product key; exploit_evidence.changed names a vulnerability, resolved to the product keys the hub holds issues for (GET /api/vulnerabilities/v1/advisories/{id}). Each page’s keys that match an indexed image of the tenant are written with the checkpoint in one transaction.
• No replay. A first start, or a new hub epoch, begins at the stream head, and envelopes the hub reclaimed below its retention horizon are logged and passed over: the timed schedules re-evaluate every image, so a missed change is late rather than lost, and the hub’s corpus import does not become a re-evaluation of every indexed package.
• Position report. After each pass the trigger reports its committed checkpoint to the hub (POST /api/vulnerabilities/v1/events/vuln.facts/consumers/{consumerId}, under the tenant’s token) with the same consumer id as its local checkpoint, jobengine.advisory-trigger-v1.tenant:{token}. The hub keeps every vuln.facts envelope above a registered position while the registration is renewed within its 24-hour lease. A refused report is logged and the pass continues; the local checkpoint stays authoritative, and a registration that stops being renewed lapses at the hub.
• Runs. Once the oldest pending change is older than DebounceSeconds (default 300), and the trigger’s previous advisory run has left planning, it creates one Advisory run on the system schedule sys-{tenant}-{TargetScheduleSlug} (default nightly-vuln-scan, analysis only), whose selector, mode and limits apply; a missing or disabled target schedule plans nothing. The planner claims up to MaxProductKeysPerRun (default 500) pending keys for the run and plans the images carrying them. Planning deletes the keys it covered; a key changed after its claim, or claimed by a run whose planning failed, returns to the pending set, and a claim older than an hour is claimed again.
• Persistence. scheduler.advisory_trigger_pending (migration 014, row-level security on tenant_id, class derived-rebuildable): one row per tenant and product key.
• Identity. The by-CVE resolution and the trigger read the hub as the dedicated client stellaops-jobengine-vulnerability-hub (advisory:read aoc:verify, secret JOBENGINE_VULNERABILITY_HUB_CLIENT_SECRET), one token per tenant, configured under Scheduler:Worker:VulnerabilityHub (Enabled, BaseAddress, Issuer, optional MetadataAddress and TokenEndpoint, ClientSecret).

Schedule limits. The planner applies a schedule’s MaxJobs before it records the run’s queued count, choosing a window over the digest-ordered images that starts at an offset derived from the run id, so successive capped runs cover different images. The runner spaces its Scanner calls so the whole run stays at RatePerSecond (each segment gets the rate divided by the shard count); Parallelism is the shard count. The seeded system scan schedules carry conservative limits: content refresh maxJobs 50, ratePerSecond 1, parallelism 1, analysis only maxJobs 200, ratePerSecond 2, parallelism 2; a system scan schedule seeded earlier with no limits at all is given them at startup, and limits an operator set are kept.

System cadence (PCD-354, owner ruling 2026-10-06). Verified against c8c4c128b5; the re-verify tests are listed below. New installations seed five schedules in UTC:

Weekly compliance sweep and reachability re-evaluation defaults are retired. Existing rows are reconciled only when their bootstrap ownership, old shipped configuration and default limits still match. Paused schedules and operator changes to cadence, selector, limits, notifications or other configuration remain intact. The tenant-bound database operation compares the observed version and configuration in the same UPDATE, so an operator edit between reading and writing wins. Retirement is a soft delete; no schedule history is removed. Repeated startup makes no further changes once the ruling has converged. Re-verify with SystemScheduleBootstrapLimitsTests and SystemScheduleReconciliationTests using tools/scripts/test-targeted-xunit.ps1.

4) Scheduler job plugins

4.1 Job-kind boundary

SchedulerJobKindCatalog (StellaOps.Scheduler.__Libraries/StellaOps.Scheduler.Plugin.Abstractions/) classifies every job kind before any registry file or mounted bundle can make it executable:

• scan is ScanJobPlugin (StellaOps.JobEngine.Scheduling/ScanJobs/ScanJobPlugin.cs).
• database-maintenance is DatabaseMaintenanceJobPlugin (StellaOps.JobEngine.Scheduling/MaintenanceJobs/). It calls each owning service’s maintenance API (Scheduler:Maintenance:Services:<n>:Name / :BaseAddress) and holds no database connection of its own.
• Scheduler:Plugins:RequiredHostJobKinds defaults to scan and database-maintenance; a missing one fails startup.
• The former doctor HostCore kind and its /api/v1/scheduler/doctor trend routes were retired on 2026-08-17; migration 010 dropped the scheduler.doctor_trends table.
• BundleRotationJob in the Worker library is not registered as a hosted service or scheduler and is not evidence of a live Attestor rotation path.

4.2 Admission of optional executable plugins

SchedulerPluginAdmission.LoadAndRegisterExternalPlugins admits optional plugins:

1. Enabled optional jobs come from Scheduler:Plugins:EnabledProfiles and Scheduler:Plugins:EnabledJobKinds; unknown kinds, unknown profiles and placeholder enablement are rejected.
2. The plugin directory (Scheduler:Plugins:Directory) is required only when an enabled optional job needs it; a missing directory then fails startup naming the job kinds.
3. Assemblies load through the shared StellaOps.Plugin.Hosting.PluginHost (src/__Libraries/StellaOps.Plugin/) with signature enforcement on by default (Scheduler:Plugins:Signature:*; EnforceSignatureVerification and AllowUnsigned cannot both be set). A discovered plugin that fails admission fails startup.
4. Only admitted ISchedulerJobPlugin types are registered, and only enabled plugins get ConfigureServices. A duplicate job kind from a bundle does not replace a HostCore handler.

Bundles sign every DLL they carry. devops/build/package-runtime-plugins.ps1 produces them under .runtime/plugins/scheduler/<profile>/ (generated, git-ignored). Compose overlays docker-compose.plugins.scheduler-ai.yml, docker-compose.plugins.scheduler-feed.yml, docker-compose.plugins.harness.yml, docker-compose.plugins.bad-signature.yml and docker-compose.plugins.scheduler-estate-doctor.yml mount them read-only into jobengine-web under /app/plugins/scheduler/<profile>, with the trust root at /app/trust-roots/plugins/scheduler. The default chain mounts no executable Scheduler bundle; jobengine-web always mounts devops/etc/plugins/scheduler (registry data, read-only) and the scheduler-plugin-scratch volume at /var/lib/stellaops/plugin-scratch/scheduler.

4.3 Estate doctor

The opt-in estate-doctor profile has two parts:

• The producer plugin StellaOps.Scheduler.Plugin.EstateDoctor (job kind estate-doctor) runs in jobengine-web. It reads estate-pack state, applies the approval gate, and queues approved dispatch envelopes. Enabling it without STELLAOPS_POSTGRES_JOBENGINE_CONNECTION fails closed.
• jobengine-estate-worker (defined in devops/compose/docker-compose.jobengine.yml, profiles: [estate-doctor]) polls the run queue, verifies each pack manifest against its signed content digest, intersects capabilities with its role (EstateDoctor:WorkerRole; unset grants nothing, an unknown name fails startup) and polls only the configured tenants (EstateDoctor:Tenants). It mounts the Docker socket (root-equivalent) and the packsregistry-upload-data volume read-only. Its build closure must never reach StellaOps.Doctor (EstateWorkerClosureConformanceTests).

A pack is keyed <manifest.packId>@<manifest.version> (packs.packs.pack_id); a manifest whose identity differs from the registry key is refused. Any change to StellaOps.JobEngine.Persistence requires re-packaging the signed producer bundle; see the estate-doctor staging runbook.

4.4 Plugin diagnostics

GET /internal/plugins/status and POST /internal/plugins/probe (StellaOps.JobEngine.Scheduling/Hosting/SchedulerPluginDiagnosticsEndpoints.cs) report job-kind registration and admission state; the probe calls ValidateConfigAsync with non-executing configuration and creates no schedules or runs. The route group is mapped AllowAnonymous(). It is an /internal/* path; treat its exposure as a network-boundary assumption, not an authorization control.

5) HTTP surface

All routes are served by jobengine-web and published to the Router gateway by service name jobengine. The wire contract is openapi/v1.json, rendered as api-reference.md(generated; do not hand-edit).

Authorization policies are registered once for every authentication branch (AddConsolidatedJobEnginePolicies in Program.cs): SchedulerPolicies.Read / .Operate / .Admin (scheduler:read / scheduler:operate / scheduler:admin), SchedulerPolicies.JobEngineRead (scheduler:read or orch:read), the Task Pack policies (§5.3), and jobengine:health:read (ops.health). Most Scheduler handlers additionally call EnsureScope(...) with a fine-grained scope. Tenant-scoped groups use RequireTenant().

5.1 Scheduler routes

Behavior notes:

• Run progress SSE. GET /api/v1/scheduler/runs/{runId}/stream emits the reconnect delay, the run snapshot, a queue-lag snapshot and heartbeats, then follows the Scheduler:RunStream poll, queue-lag and heartbeat cadences until the run is terminal. It carries progress state, not run logs.
• Worker visibility. GET /api/v1/scheduler/workers takes an optional limit (1–100, default 20). The response separates the declared execution mode (embedded/standalone, from Scheduler:Worker:Embedded) from registration visibility, so an empty registry in embedded mode is not reported as “no execution”. Process ids and free-form metadata are not returned.

5.2 JobEngine compatibility read surface

Group policy SchedulerPolicies.JobEngineRead. These routes project durable Scheduler Run rows; they never fabricate placeholder jobs.

• GET /api/v1/jobengine/jobs — tenant-scoped records of jobType scheduler-run; any other requested jobType returns an empty list.
• GET /api/v1/jobengine/jobs/{jobId} and GET /api/v1/jobengine/jobs/{jobId}/detail — the detail payload is the canonical jobengine.scheduler-run.evidence.v1 envelope. Proof objects appear only when present in scheduler.runs.reason; absent proofs report available=false with a reason. payloadDigest is sha256: over the returned canonical payload.
• GET /api/v1/jobengine/dag/job/{jobId}/parents and /children — edges derived only from scheduler.runs.retry_of.

No other service maps an /api/v1/jobengine/* prefix. Release Orchestrator’s former aliases (runs, audit, slos, deadletter, dag, pack-runs, stream, sources) are removed; its real capabilities are served at /api/v1/release-orchestrator/runs/{runId}/first-signal, /api/v1/release-orchestrator/audit and /api/v1/release-orchestrator/slos.

5.3 Task Pack registry routes

Composed by src/JobEngine/StellaOps.JobEngine.WebService/PacksRegistryComposition.cs. Every /api/v1 route carries one of two policies (PacksRegistryPolicies):

• The tenant comes only from the validated stellaops:tenant claim. A body or query tenant that differs returns 403 tenant_claim_mismatch; a missing claim returns 400 tenant_missing. Reads are keyed (tenant_id, pack_id) and return the same 404 whether a pack is absent or belongs to another tenant.
• PacksRegistry:Auth:ApiKey (X-API-Key) is an optional supplementary gate, compared in constant time; it grants nothing on its own. PacksRegistry:Auth:AllowedTenants is a coarse allow-list.
• There is no approval endpoint and no version-listing endpoint; GET /api/v1/packs lists packs.
• The generated /openapi/v1.json is the only pack-registry contract URL. The former /openapi/packs.json and /openapi/pack-manifest.json redirects were removed.

5.4 Health, discovery and Doctor

• GET /healthz, GET /readyz, MapOpenApi() at /openapi/v1.json, and the build-info endpoints (/api/v1/buildinfo, /buildinfo.json, fallback module name jobengine) are anonymous.
• GET /doctor/jobengine-web/checks requires jobengine:health:read (ops.health). It runs the standard database checks (doctor.standard.db.connection, .migration-status, .size-budget), the Valkey owned-state size checks for the Scheduler planner/runner streams (registered when scheduler:queue:Kind is Redis), and doctor.jobengine.retention.terminal-budgets. The PostgreSQL host also registers doctor.jobengine.capacity.budget over its owner-local receipt. See capacity monitoring and its limits.
• Registration with the Platform capability registry is controlled by Doctor:Registration:* (Enabled defaults to false in compose via JOBENGINE_DOCTOR_REGISTRATION_ENABLED). When enabled, Authority, ClientId and ClientSecret must be non-blank or startup fails.
• jobengine-worker and jobengine-estate-worker expose only /health/liveness and /health/readiness; their database is covered by the web host’s checks.

6) Task Pack storage contract

The contract below is binding for the Task Pack registry. The pre-consolidation PacksRegistry dossier that carried the long-form rationale is archived at docs-archive/modules/packsregistry/architecture.md; where it disagrees with this section or with src/, this section and src/ win. Operator procedure: PacksRegistry dual-mode storage runbook.

6.1 Stores and configuration

jobengine-web runs with PacksRegistry:Storage:Driver=postgres and PacksRegistry:Storage:ObjectStore:Driver=dual-fs:

• The two roots must be distinct and non-overlapping; canonical-path and reparse/symlink checks fail startup otherwise. Production proves the seed mount is read-only and requires a local Linux filesystem; network-backed exports fail closed.
• Driver=seed-fs is a seed-only posture: blob-writing mutations return 503 packs_upload_store_unavailable. Storage:Driver=filesystem is a Development/Testing harness and is rejected in Production.
• Capacity: PacksRegistry:Storage:ObjectStore:Capacity:MaxPackBytes, MaxTenantBytes, MaxTenantObjects, MaxTotalBytes, MaxTotalObjects, MinimumFreeBytes; also OrphanGracePeriod and Seed:MaxManifestBytes / Seed:MaxManifestEntries. A full store returns 507 packs_upload_capacity_exhausted; filesystem faults are 503/500, never an authentication failure.
• Pack signature verification: PacksRegistry:Verification:* (PublicKeyPem, PublicKeyPemFile, TrustSetJson, TrustEpoch). PublicKeyPemFile names a file that holds the RSA public key; the host reads it into PublicKeyPem before the startup guard and refuses a missing or empty file, or a file named while PublicKeyPem already has a value. The dev estate compose runs Development with PacksRegistry:Verification:LocalHarness:Enabled=true. The release bundle runs Production: each installation generates its packs-signing RSA key pair, mounts the public half at /app/etc/jobengine/keys/packs-signing.pub.pem (PublicKeyPemFile), and signs the shipped empty seed generation with the private half, which is never mounted.
• Seed manifest signature: base64 RSA PKCS#1 v1.5 SHA-256 over the UTF-8 text sha256:<hex digest of the manifest's canonical bytes> (the manifest without signature, keys sorted, no whitespace). Outside Development and Testing an unsigned or unverifiable manifest quarantines the generation and startup fails.
• Pack content is data only. Import never executes pack-supplied code, and pack content is never mounted under /app/plugins.

6.2 Identity, origin and collisions

• Logical identity is (tenant_id, pack_id); pack_id is derived from normalized name and version by one canonicalizer shared by ingest, seed reconciliation, persistence and export.
• Blob identity is the verified SHA-256 digest and kind (content, provenance, attestation) under the tenant prefix. User text never becomes a path component.
• Origins are legacy-inline, seed, upload and seed+upload; they appear in list, manifest, audit and export output. The same identity and digest is an idempotent duplicate; a ready seed row plus a verified identical upload is the one permitted change (to seed+upload).
• The same identity with a different digest returns 409 pack_identity_digest_conflict for an API request, or quarantines the seed generation (readiness fails).
• Seed bytes are immutable: no mutation writes or deletes in the seed store. Seed generations are append-only evidence; an active generation that disappears from the mount is quarantined, not tombstoned.

6.3 Commit order and recovery

Uploads validate tenant, scope, shape, identity, digests and signature first; then, under the shared storage-admission advisory lock, claim a staging row; then reserve capacity and write blobs atomically through .control/upload-temps/v1/<shard> on the upload volume; then, in one transaction, mark the row ready and append the storage event, outbox entry and mutation audit. Stale-claim cleanup uses database time, the configured grace period and a reference proof, and never collects a seed+upload final artifact.

6.4 Signature floor

Each (tenant_id, region_profile) has a durable signature floor (active key id plus observed trust epoch, with every observation kept in packs.pack_signature_floor_epochs). A stale key receives 409 pack_signature_floor_mismatch; an attempt to roll the floor back receives 409 pack_signature_floor_regression; a claim revived under a different key receives 409 pack_upload_claim_verification_mismatch; floor advance during in-flight claims returns 409 re_envelope_claims_active.

6.5 Legacy upgrade gate

An estate that ran the pre-dual-mode binary maps its old mutable payload path to PacksRegistry:Storage:ObjectStore:LegacySeedFs:RootPath. The upgrade is non-rolling: a read-only preflight inventory must pass before any DDL, after which a durable transition journal stages artifacts into the upload store. A crash leaves an explicit transition-in-progress state that resumes from the journal. LegacyPayloadMode and LegacyUpgradeMaintenanceMode under PacksRegistry:Storage:ObjectStore control the posture.

7) Offline-seed export and import

• Export (POST /api/v1/export/offline-seed, write policy) streams one tenant-scoped, repeatable-read snapshot as a ZIP. When a complete byte set is requested it includes catalog-manifest.json; with an export signing key configured it adds the DSSE envelope catalog-manifest.json.dsse.json. includeContent and includeProvenance default to false.
• Signing key (PacksRegistryExportSigning.cs): an ES256 key from STELLAOPS_PACKSREGISTRY_EXPORT_ES256_PRIVATE_KEY_PEM_FILE (compose mounts devops/etc/jobengine read-only at /app/etc/jobengine; devops/compose/scripts/ensure-dev-certs.sh generates the per-lab key under the git-ignored devops/etc/jobengine/keys/), or the PacksRegistry:Export:Signing:* keys. Without a key the manifest ships unsigned.
• Key id (OSR-1, 2026-09-26): unless STELLAOPS_PACKSREGISTRY_EXPORT_KEY_ID pins one, the stamped keyid is derived from the key as sha256-<lowercase hex SHA-256 of the DER SubjectPublicKeyInfo>, so relabelling the same key is impossible and rotating it changes the id. Rotation: devops/compose/scripts/rotate-jobengine-export-key.sh, then publish the public half with stella crypto verification-key introduce. Procedure: pack-signing-and-rbac.md §7.1.
• Import (POST /api/v1/import/offline-seed, write policy) verifies the archive with the same SDK as stella verify offline-seed (StellaOps.PacksRegistry.OfflineSeed.Contracts) and imports only a VERIFIED archive with no findings, writing each pack through the ordinary upload path into the caller’s claim tenant. Refusals write an offline.seed.import.refused audit row and nothing to the catalog. The trust root is PacksRegistry:Export:Verification:* (the ES256 export key family, separate from the RSA pack-signing root); with none configured the endpoint returns 503.
• Anchoring (PacksRegistry:Import:Anchoring:*): each admitted import’s envelope is sent to the Attestor transparency log over HTTP; when the log is unreachable the import is still admitted and the anchor is queued in packs.import_anchors.

8) Runtime configuration

• Authentication. Three branches, all registering the same policies: Scheduler:Authority when Enabled=true; otherwise the canonical Authority:ResourceServer section (the deployed branch; compose sets Authority__ResourceServer__Authority and the Router metadata address); otherwise header-based local harness auth (TestingLocalHarness, or Development/Testing with Scheduler:LocalHarness:Enabled=true). Outside Development/Testing a missing Authority:ResourceServer:Authority fails startup. Scheduler:Authority:AllowAnonymousFallback is refused outside the harness modes. UseIdentityEnvelopeAuthentication() runs before UseAuthentication() so gateway-forwarded identity envelopes are accepted.
• Embedded worker. Scheduler:Worker:Embedded (default true; compose sets true) runs the planner, runner, policy-dispatch, graph build/overlay and trigger services in jobengine-web, and only when the database is configured. ExceptionLifecycleWorker, ExpiringNotificationWorker and SystemScheduleBootstrap run in the web process in either mode. Other worker classes in StellaOps.Scheduler.Worker (resolver, reachability, policy re-evaluation/reconciliation, simulation reduction) are not registered as hosted services by this host.
• Queue. scheduler:queue (Kind = Redis or Nats; compose uses Redis on cache.stella-ops.local:6379), including MaxDeliveryAttempts; HLC ordering under Scheduler:Queue:Hlc (DSSE signing under :DsseSigning).
• Notifications. Worker events and exception notifications publish through the Notify event queue (notify:queue) via StellaOps.Notify.EventClient and StellaOps.Notify.Contracts. Missing wiring fails closed outside harnesses (Scheduler:Worker:Notifications:UseLocalHarness=true in Development).
• By-CVE resolution. ResolveByVulnerabilitiesAsync resolves vulnerability ids to the product keys the Vulnerabilities hub holds consensus issues for (GET /api/vulnerabilities/v1/advisories/{id}, HubVulnerabilityPurlResolver), with the hub identity of the advisory trigger. An unknown id resolves to nothing; a hub outage fails the call, and policy-run targeting then falls back to a full run. With Scheduler:Worker:VulnerabilityHub:Enabled false the resolver is empty. Compose enables it.
• Shared token client. Scheduler:Worker:Concelier:Authority registers the host’s shared client-credentials token client as stellaops-scheduler-concelier-observations (scope ops.health, secret SCHEDULER_CONCELIER_AUTHORITY_CLIENT_SECRET); the database-maintenance job authenticates through it. The section keeps its historical name; the Concelier resolver it served was retired.
• Content refresh. In content-refresh mode the runner first submits a rescan (POST /api/v1/scans) with the image’s indexed location (registry/repository@digest) and its digest. An image the index holds only under the unresolved placeholder has no location; the runner logs a warning, submits no rescan and still requests the report. Scanner cannot pull a digest alone and fails such an image scan, so a digest-only rescan cannot succeed with an empty inventory and empty the image’s components in the index.
• Scanner runner identity. The runner’s Scanner calls (POST /api/v1/reports, and POST /api/v1/scans before it in content-refresh mode) authenticate as the dedicated client stellaops-scheduler-scanner-runner with scopes scanner:read scanner:scan, configured under Scheduler:Worker:Runner:Scanner:Authority (Enabled, Issuer, optional MetadataAddress and TokenEndpoint, ClientSecret; compose supplies the secret from SCHEDULER_SCANNER_RUNNER_CLIENT_SECRET, which Authority’s Standard descriptor also reads). The client id and scopes are fixed in code. Each request names the run’s tenant in X-StellaOps-TenantId, and the runner mints one token per tenant, because Scanner reads the tenant from the token claim. The token client is built in its own container, separate from the host’s shared Concelier client. With the identity disabled (the default outside compose) every Scanner call is refused before it is sent. Enabled without an Issuer or ClientSecret, the host fails to start and names the missing key.
• Other sections. Scheduler:Events (graph-job completion events), Scheduler:Cartographer (a disabled webhook resolves a client that makes no outbound call), Scheduler:RunStream, Scheduler:Plugins, Scheduler:Maintenance, Scheduler:Worker:* base addresses (Scanner, Cartographer/graph, Policy, JobEngine), JobEngine:Retention, PacksRegistry:* (§6, §7), Doctor:Registration:*, and Router.

9) Gateway routing

jobengine-web registers with the Router as microservice jobengine (AddRouterMicroservice(serviceName: "jobengine"), consumer group jobengine) and publishes its endpoints through the HELLO handshake. The gateway carries no static route rows for JobEngine: the former /api/v1/jobengine/jobs, /api/v1/jobengine/dag, /api/v1/jobengine/registry/packs, /api/v1/scheduler/jobs, /api/scheduler, /scheduler host-prefix and /api/v1/scheduler/doctor rows are listed as retired in src/Router/__Tests/StellaOps.Gateway.WebService.Tests/Configuration/retired-routes.json, and re-adding one is a regression. Paths JobEngine does not map (for example /api/v1/jobengine/runs) are published by no service (§5.2).

10) Observability

Metric instruments defined in the Scheduler libraries include scheduler_queue_depth, scheduler_queue_enqueued_total, scheduler_queue_deduplicated_total, scheduler_queue_ack_total, scheduler_queue_retry_total, scheduler_queue_deadletter_total, scheduler_runner_backlog, scheduler_hlc_enqueues_total, scheduler_hlc_enqueues_duplicates_total, scheduler_hlc_enqueue_latency_ms, scheduler_chain_verifications_total, scheduler_chain_verification_failures_total, scheduler_batch_snapshots_total, graph_jobs_inflight, graph_build_seconds, overlay_lag_seconds, policy_simulation_queue_depth and policy_simulation_latency. Re-verify with rg -n "scheduler_|graph_|overlay_|policy_simulation_" src/JobEngine --glob "*.cs".

Audit events are emitted through AddAuditEmission (the release bundle sets AuditEmission__TimelineBaseUrl). The generic job metrics (job_queue_depth, job_latency_seconds, pack_run_*) are not emitted by this module.

11) Deployment wiring

• Estate compose (devops/compose/docker-compose.stella-services.yml, slot 19): jobengine-web (image stellaops/jobengine-web, aliases jobengine.stella-ops.local and jobengine-web.stella-ops.local, loopback 127.1.0.19:80) and jobengine-worker (image stellaops/jobengine-worker, alias jobengine-worker.stella-ops.local). Both require STELLAOPS_POSTGRES_JOBENGINE_CONNECTION. jobengine-estate-worker is in devops/compose/docker-compose.jobengine.yml behind the estate-doctor profile.
• Release bundle (devops/release/bundle/docker-compose.yml): jobengine-web and jobengine-worker pulled by ${STELLA_VERSION} from ${STELLA_REGISTRY}, with bind mounts ./config/etc/jobengine, ./config/etc/plugins/packsregistry and ./config/etc/plugins/scheduler (read-only), and named volumes packsregistry-upload-data (/var/lib/stellaops/packsregistry/uploads) and scheduler-plugin-scratch (/var/lib/stellaops/plugin-scratch/scheduler). The bundle does not include jobengine-estate-worker.
• Provisioning. tools/scripts/deploy/postgres/provision-service-database.sh stellaops_jobengine jobengine creates the database and role and applies the REVOKE CONNECT boundary. The service then migrates itself on startup.

12) Worker SDKs (orphaned)

src/JobEngine/StellaOps.JobEngine.WorkerSdk.Go/ and src/JobEngine/StellaOps.JobEngine.WorkerSdk.Python/ implement a claim/ack/heartbeat/progress client for POST /api/jobs/lease, /api/jobs/{jobId}/ack, /heartbeat and /progress. No service in this repository serves /api/jobs/*; only the SDKs and their self-mocking tests reference it. They are the client half of the removed TaskRunner/legacy orchestrator and are not Release Orchestrator’s agent protocol either. They are not packaged or deployed. Do not build against them. The Go SDK still runs in the report-only .gitea/workflows/go-tests.yml lane against its own mock. Removal is recorded as an open dead-code item (SPRINT_20260712_009 CLO-5, archived).

13) History

• The old StellaOps.JobEngine.* libraries (Core, Infrastructure, Worker, Tests) and the TaskRunner service were removed in April 2026. No task_runner_id column exists in the scheduler schema.
• ADR-039 D14 merged the Scheduler and PacksRegistry deployables into this family (SPRINT_20260722_012, archived): JOB-9 (2026-09-12) cut over the live estate, moved both DALs into StellaOps.JobEngine.Persistence and froze the predecessor hosts; JOB-10 (2026-09-13) froze the worker hosts and retired the soak aliases; JOB-11 to JOB-13 delivered and exercised the dual-mode Task Pack storage. The design record is consolidation-design.md.
• The pre-consolidation Scheduler and PacksRegistry dossiers are archived under docs-archive/modules/scheduler/ and docs-archive/modules/packsregistry/. Their directories in docs/modules/ are pointer stubs.

Related documents

• README — module overview.
• Retention and custody.
• Estate-doctor plugin staging.
• PacksRegistry dual-mode storage runbook.
• Pack signing and RBAC.
• Release Orchestrator — the release control plane.