Scheduler Architecture

Source: src/JobEngine/StellaOps.Scheduler.* Parent dossier: docs/modules/jobengine/architecture.md

Scheduler is the JobEngine subdomain that plans and executes recurring or event-driven jobs for already-cataloged Stella Ops data. It owns the scheduler PostgreSQL schema, exposes /api/v1/scheduler/**, and runs the embedded worker services by default.

Resource-server authentication discovery

Scheduler validates restricted bearer tokens against the public Authority issuer. In the canonical compose stack, scheduler-web retrieves OpenID metadata and JWKS through the TLS-terminating Router at https://router.stella-ops.local/.well-known/openid-configuration. It must not use direct http://authority.stella-ops.local discovery: Authority rejects OpenID discovery over HTTP, which prevents bearer validation and turns otherwise valid restricted-JWT reads into service-unavailable responses.

HTTPS discovery also requires the Router certificate chain. Scheduler mounts the canonical ${STELLAOPS_CA_BUNDLE_VOLUME} read-only at /etc/ssl/certs/ca-certificates.crt and sets SSL_CERT_FILE to that exact target. devops/compose/scripts/ensure-dev-certs.sh maintains the source combined-ca-bundle.crt by replacing its marked local development certificate entry. Omitting either the mount or SSL_CERT_FILE leaves the HTTPS metadata client unable to trust the Router and produces another pre-endpoint service-unavailable failure.

The adjacent Scheduler WebService compose contract test reads the canonical scheduler-web service block, requires exactly this HTTPS Router value plus the trust environment/mount, and rejects the obsolete direct HTTP Authority value. Append-only docs/qa/feature-checks/runs/web/scheduler-access-runtime-truth/run-013/tier2-api-check.json is the authoritative pre-fix 503 evidence. These source invariants do not prove a running container until the corrected compose configuration has been deployed under explicit approval.

Run Progress SSE

GET /api/v1/scheduler/runs/{runId}/stream is the tenant-scoped, scheduler:read-protected run-progress stream. It immediately emits the reconnect delay, initial run snapshot, queue-lag snapshot, and heartbeat. Non-terminal runs then use the configured Scheduler:RunStream poll, queue-lag, and heartbeat cadences; state changes, progress, deltas, disappearance, and terminal completion retain their existing named SSE events.

Each cadence owns exactly one active PeriodicTimer.WaitForNextTickAsync operation. The coordinator rearms only the cadence that completed, so an active stream remains valid across repeated poll boundaries without overlapping waits on the same timer. Expected authorization failures are classified before streaming begins. An internal coordinator InvalidOperationException is not converted to HTTP 403, because a runtime failure must not masquerade as missing permission.

This endpoint carries run-progress state, not run logs. The Console can therefore truthfully report that logs are unavailable even when direct Scheduler progress SSE works. The current Router ASP.NET bridge defaults discovered endpoints to SupportsStreaming=false and does not forward live response chunks, so this direct-service invariant does not claim real-time frontdoor delivery; that is a separate Router dependency.

Worker execution visibility

GET /api/v1/scheduler/workers is the bounded execution-registration read for the Operations dashboard and Scheduler diagnostics. It requires an authenticated tenant plus scheduler:read (the handler’s scheduler.workers.read child scope is satisfied through the canonical scope hierarchy). The optional limit is 1 through 100 and defaults to 20.

The scheduler.workers table is intentionally global and is not protected by tenant RLS, so the repository uses a system connection and applies the exact caller boundary in SQL: the caller’s dedicated registrations plus registrations whose tenant_id is null and therefore explicitly shared. PostgreSQL computes exact total/active/stale/idle counts, orders by heartbeat then worker id, and applies the response cap before rows leave storage. Process ids and free-form worker metadata remain internal.

Execution topology and registration visibility are deliberately separate. The response states executionMode=embedded|standalone and reports an embedded host as available even when the registration table is empty; an empty registry is therefore visibilityState=no_registrations, not fabricated proof that no work can execute. In standalone mode, zero active registrations is explicitly executionAvailability=unavailable. This read supports the existing Dashboard, Schedules/Runs, and Doctor drill-downs; it does not restore the retired generic Workers screen.

Job Code Boundary

Scheduler job identifiers are classified in source before any registry or mounted bundle can make them executable:

The source catalog is SchedulerJobKindCatalog in StellaOps.Scheduler.Plugin.Abstractions. It is the authority for whether a job kind can attach executable plugin code. Registry YAML remains data/config; it cannot convert a declarative or placeholder entry into code.

Built-In Handlers

The Scheduler host registers built-in handlers directly from host-owned source:

• scan via ScanJobPlugin in StellaOps.Scheduler.WebService
• doctor via SchedulerDoctorJobPlugin in StellaOps.Scheduler.WebService

Startup requires those handlers to resolve from the in-process registry. A missing built-in fails the host at startup (RequireExecutableJobKinds throws during builder.Build(), Program.cs:292) — the process never reaches the readiness probe rather than reporting unready, so a broken host does not serve at all. These built-ins do not require mounted plugin signatures because they are part of the host binary and release evidence.

Optional Executable Plugins

Optional executable jobs are admitted only through SchedulerPluginAdmission:

1. Resolve enabled optional jobs from Scheduler:Plugins:EnabledProfiles and Scheduler:Plugins:EnabledJobKinds.
2. Reject unknown job kinds, unknown profiles, and placeholder enablement.
3. Require the plugin directory only when an enabled optional job needs a mounted bundle.
4. Load mounted plugin assemblies with PluginHostOptions.EnforceSignatureVerification=true by default.
5. Fail startup when any discovered plugin fails signature/load admission.
6. Register only admitted ISchedulerJobPlugin types, and call ConfigureServices only for enabled plugins.

Disabled optional plugins may be visible in the registry descriptors, but Resolve(jobKind) returns null until the plugin is enabled and signature admission has succeeded.

The deterministic harness bundle StellaOps.Scheduler.Plugin.ExternalEcho implements third-party-job for the scheduler-external profile. It exists to prove signed mounted scheduler job admission without promoting a production AI, feed, or maintenance adapter. The packager emits it only for harness and bad-signature profiles under devops/plugins/scheduler/<profile>/stellaops.scheduler.plugin.external-echo/. The bad-signature profile deliberately corrupts the detached signature for negative admission coverage.

Signature Contract

An executable scheduler plugin bundle must provide:

• a stable job kind that appears in SchedulerJobKindCatalog as OptionalExecutablePlugin;
• a profile or explicit job-kind enablement in Scheduler configuration;
• a plugin assembly matching the configured discovery pattern;
• a successful signature verification result from the configured verifier;
• capabilities and dependencies declared in the source descriptor and mirrored by bundle metadata in the packaging layer.

Scheduler package manifests mirror the admitted job boundary with jobKinds, capabilities, dependencies, version, and an assembly entry containing the payload SHA-256 plus detached signature path. Registry YAML remains operator-editable data/config and cannot activate unsigned executable code by itself.

Scheduler runtime bundles sign every DLL copied into the bundle directory, not only the primary plugin assembly. The loader verifies discovered DLL payloads fail-closed; a copied dependency without a detached signature prevents the optional profile from becoming ready.

The current host supports the shared StellaOps.Plugin loader and optional Cosign verifier wiring. Compose mounts Scheduler plugin bundles read-only only for explicit optional executable Scheduler profiles from devops/plugins/scheduler/<profile>/<plugin-id>/ to /app/plugins/scheduler/<profile>/<plugin-id>/, with config at /app/etc/plugins/scheduler, trust roots at /app/trust-roots/plugins/scheduler, and scratch/probe output under /var/lib/stellaops/plugin-scratch/scheduler.

Default Directory Behavior

The Scheduler no longer creates plugins/scheduler just because the service started. Missing optional plugin directories are tolerated when no optional executable job is enabled. If a profile such as scheduler-ai is enabled and the plugin directory is missing, startup fails with the enabled job kinds named in the error.

Base and recommended compose overlays do not declare Scheduler executable plugin mounts. The harness and bad-signature overlays opt in to the generated scheduler-external ExternalEcho bundle, set recursive discovery, mount /app/plugins/scheduler/<profile> and the scheduler trust root read-only, and enable only the third-party-job optional executable profile. The production optional overlays docker-compose.plugins.scheduler-ai.yml and docker-compose.plugins.scheduler-feed.yml follow the same mounted-bundle shape for scheduler-ai and scheduler-feed respectively. Core jobs and registry data remain host-owned/declarative and are not represented by empty bundle roots.

Plugin Probe Diagnostics

Scheduler WebService exposes the canonical runtime-composition diagnostics:

• GET /internal/plugins/status returns the current Scheduler job plugin discovery/admission/load state without executing plugin work.
• POST /internal/plugins/probe accepts the shared PluginProbeRequest shape and runs deterministic probes over selected plugins or all visible executable Scheduler plugin descriptors.

Both /internal/plugins/* routes are mapped without an explicit scope policy (Hosting/SchedulerPluginDiagnosticsEndpoints.cs); they are /internal/* paths with no gateway route, so they are reachable only in-cluster. That is a network-boundary assumption, not an authorization guarantee — do not expose them through a gateway route without adding a scope gate.

The probe uses the real ISchedulerJobPlugin boundary by calling ValidateConfigAsync with deterministic probe config. Host-owned handlers use an empty config. advisory-ai-analysis receives a non-executing advisory summary config, and feed-processor receives a non-executing feed validation job-kind config. It does not create schedules, enqueue runs, or execute jobs. Host-owned handlers report the base profile. Disabled optional plugin descriptors remain visible as disabled or not-mounted until a signed mounted bundle is enabled by profile/job-kind configuration. Acceptance treats scan and doctor as required executable surfaces. Disabled optional descriptors and declarative job identifiers remain visible for diagnostics but are not blocking unless an explicit optional executable Scheduler profile or job-kind allowlist enables them.

Current Status

The existing devops registry under devops/etc/plugins/scheduler/registry.yaml is still a data/config registry and currently lists entries that the source catalog treats as host-owned, declarative, optional, or placeholder. The harness profile has signed bundle production for third-party-job. The production optional profiles now have signed bundle producers for advisory-ai-analysis (scheduler-ai) and feed-processor (scheduler-feed) plus static compose overlays. Local release closeout evidence proves both production optional profiles are mounted, admitted, loaded, and probed through /internal/plugins/probe, and the mounted-mode stellaops/scheduler-web:dev image audit proves optional Scheduler plugin payloads are not baked into the service image.