Estate-doctor Scheduler/worker plugin staging

CURRENCY — 2026-09-12 (SPRINT_20260722_012 JOB-9). The producer host in this runbook is now jobengine-web, not scheduler-web: the ADR-039 D14 cutover merged scheduler-web and packsregistry-web into the consolidated JobEngine family and froze both predecessors under src/__Obsoleted/JobEngine/. Two consequences for the steps below. (a) Every service name, pin file and build key was repointed in place (docker-compose.local-jobengine-web.yml succeeds docker-compose.local-scheduler-web.yml). (b) The former step 9 (“Seed stellaops_platform”) was deleted on 2026-09-14 (SPRINT_20260914_003 JEF-1). It existed only because the live scheduler fired from stellaops_platform while the producer read constraints and pack content from stellaops_jobengine, so the same schedule had to be seeded into BOTH. After the cutover there is ONE database — seed stellaops_jobengine (step 7) and stop; PLT-4 dropped the platform-side scheduler/packs schemas on 2026-09-14. The simple-identifier rule it carried now lives in step 7: it is a model-level read validation that applies to whatever you write. Paragraphs that record what was MEASURED on 2026-09-10 keep their original service names on purpose; they are history, not instructions.

Status and boundary

The first part of this runbook is repository-side staging: it prepares the signed Scheduler producer and exact five-family Doctor runtime-plugin set, and proves that their held-back Compose overlay renders. It does not by itself authorize a service start, recreate, deployment, image or registry publish, release, or live-estate forcing call. The estate activation record at the end is the procedure that WAS executed on the lab estate on 2026-09-10 under an owner-directed window (SPRINT_20260722_009 DOC-4b); it is the reference for the next estate.

The overlay devops/compose/docker-compose.plugins.scheduler-estate-doctor.yml is absent from the default Compose chain. It takes effect only when an owner-attended window explicitly adds the file and selects --profile estate-doctor.

Verified against implementation commits c8efe1397e80161f5abb625b159aceac894e6b62, 1fa385c0b53b70dceefd232f72ff4df245270e76, and ab32c14569547aaca53de9c69366d0756fb07657. Re-verify the contract and admission behavior with the targeted commands in Repository verification.

Artifact map

PlaneStaged artifactAdmission contract
Scheduler producerdevops/plugins/scheduler/estate-doctor/stellaops.scheduler.plugin.estate-doctor/id stellaops.scheduler.plugin.estate-doctor; capability scheduler.job.estate-doctor; job kind/profile estate-doctor
Estate worker checksdevops/plugins/doctor/estate-doctor/exactly five child bundles: stellaops.doctor.docker, stellaops.doctor.observability, stellaops.doctor.servicegraph, stellaops.doctor.security, stellaops.doctor.core.worker; capability doctor:checks; every managed DLL has its own detached signature
Scheduler trust rootdevops/etc/certificates/trust-roots/plugins/scheduler/cosign.pubmounted read-only at /app/trust-roots/plugins/scheduler/cosign.pub
Doctor trust rootdevops/etc/certificates/trust-roots/plugins/doctor/cosign.pubmounted read-only at /app/trust-roots/plugins/doctor/cosign.pub

The Scheduler bundle is only the producer that evaluates the pack gate and queues approved work. It deliberately contains no Doctor implementation DLL. The separate Doctor root contains exactly the five ruled estate families and no other executable family. The loader binds each id to its exact primary assembly and refuses a missing or unexpected directory before loading any assembly. The overlay mounts both bundle roots and both trust roots read-only; it does not alter the base worker’s Docker endpoint or socket mapping.

Required operator inputs

Supply secrets through the operator’s environment or mounted files. Never put their values in this repository, command history, chat, or evidence output.

Stage the signed bundles

Run these from the repository root only after the signing key paths and public key paths are present in the local operator environment:

pwsh ./devops/build/package-runtime-plugins.ps1 `
  -Module scheduler `
  -Profile estate-doctor `
  -SignSchedulerBundles `
  -SchedulerCosignKeyPath $env:SCHEDULER_COSIGN_KEY_PATH `
  -SchedulerCosignPublicKeyPath $env:SCHEDULER_COSIGN_PUBLIC_KEY_PATH

pwsh ./devops/build/package-runtime-plugins.ps1 `
  -Module doctor `
  -Profile estate-doctor `
  -SignDoctorBundles `
  -DoctorCosignKeyPath $env:DOCTOR_COSIGN_KEY_PATH `
  -DoctorCosignPublicKeyPath $env:DOCTOR_COSIGN_PUBLIC_KEY_PATH

The 2026-08-26 repository rehearsal used only an ephemeral scratch RSA key under git-ignored tmp/scratch/; no production signing key was used and no artifact was published or activated. Generated bundles, signatures, reports, and copied trust roots are operational material: inspect them locally and never commit them.

Inspect the staged artifacts

The canonical descriptor is manifest.json (not runtime-plugin.json). The following checks must all pass before a window can be considered:

$schedulerBundle = 'devops/plugins/scheduler/estate-doctor/stellaops.scheduler.plugin.estate-doctor'
$doctorRoot = 'devops/plugins/doctor/estate-doctor'
$doctorFamilies = [ordered]@{
  'stellaops.doctor.docker' = 'StellaOps.Doctor.Plugins.Docker.dll'
  'stellaops.doctor.observability' = 'StellaOps.Doctor.Plugins.Observability.dll'
  'stellaops.doctor.servicegraph' = 'StellaOps.Doctor.Plugins.ServiceGraph.dll'
  'stellaops.doctor.security' = 'StellaOps.Doctor.Plugins.Security.dll'
  'stellaops.doctor.core.worker' = 'StellaOps.Doctor.Plugins.Core.Worker.dll'
}

$requiredDoctorIds = @($doctorFamilies.Keys | Sort-Object)
$actualDoctorIds = @(Get-ChildItem -LiteralPath $doctorRoot -Directory |
  Select-Object -ExpandProperty Name | Sort-Object)
if (@(Compare-Object $requiredDoctorIds $actualDoctorIds).Count -ne 0) {
  throw "The estate-doctor Doctor profile must contain exactly: $($requiredDoctorIds -join ', ')"
}

$schedulerManifest = Get-Content -LiteralPath (Join-Path $schedulerBundle 'manifest.json') -Raw | ConvertFrom-Json
if ($schedulerManifest.id -ne 'stellaops.scheduler.plugin.estate-doctor' -or
    $schedulerManifest.scheduler.jobKind -ne 'estate-doctor' -or
    $schedulerManifest.scheduler.enabledProfile -ne 'estate-doctor') {
  throw 'The Scheduler estate-doctor manifest is not the expected producer.'
}

if (Get-ChildItem -LiteralPath $schedulerBundle -Filter 'StellaOps.Doctor*.dll' -File -Recurse) {
  throw 'The Scheduler producer bundle must not carry Doctor implementation DLLs.'
}

foreach ($family in $doctorFamilies.GetEnumerator()) {
  $doctorBundle = Join-Path $doctorRoot $family.Key
  $manifest = Get-Content -LiteralPath (Join-Path $doctorBundle 'manifest.json') -Raw | ConvertFrom-Json
  $capabilities = @($manifest.capabilities)
  if ($manifest.id -ne $family.Key -or $manifest.module -ne 'doctor' -or
      $manifest.profile -ne 'estate-doctor' -or
      $manifest.contractVersion -ne 'runtime-bundle.v1' -or
      $manifest.assembly.path -ne $family.Value -or
      $capabilities.Count -ne 1 -or $capabilities[0] -ne 'doctor:checks') {
    throw "Doctor manifest binding is invalid for $($family.Key)."
  }

  $primary = Join-Path $doctorBundle $family.Value
  $actualPrimaryHash = (Get-FileHash -LiteralPath $primary -Algorithm SHA256).Hash.ToLowerInvariant()
  if ($actualPrimaryHash -ne ([string]$manifest.assembly.sha256).ToLowerInvariant()) {
    throw "Primary digest mismatch for $($family.Key)."
  }

  $unsignedDoctorDlls = @(Get-ChildItem -LiteralPath $doctorBundle -Filter '*.dll' -File -Recurse |
    Where-Object { -not (Test-Path -LiteralPath ($_.FullName + '.sig') -PathType Leaf) })
  if ($unsignedDoctorDlls.Count -ne 0) {
    throw "Every managed Doctor DLL must have a detached signature; missing: $($unsignedDoctorDlls.FullName -join ', ')"
  }

  foreach ($line in Get-Content -LiteralPath (Join-Path $doctorBundle 'checksums.sha256')) {
    if ($line -notmatch '^([0-9a-f]{64})  (.+)$') {
      throw "Malformed checksum row in $($family.Key): $line"
    }
    $payload = Join-Path $doctorBundle ($Matches[2] -replace '/', [IO.Path]::DirectorySeparatorChar)
    $actual = (Get-FileHash -LiteralPath $payload -Algorithm SHA256).Hash.ToLowerInvariant()
    if ($actual -ne $Matches[1]) {
      throw "Checksum mismatch in $($family.Key): $($Matches[2])"
    }
  }
}

Render the held-back Compose profile

Rendering proves merge/configuration coherence only. It does not start or recreate any container.

Windows PowerShell uses NUL as the empty env file:

docker compose --env-file NUL `
  -f devops/compose/docker-compose.stella-services.yml `
  -f devops/compose/docker-compose.jobengine.yml `
  -f devops/compose/docker-compose.plugins.scheduler-estate-doctor.yml `
  --profile estate-doctor config --no-interpolate --quiet
if ($LASTEXITCODE -ne 0) { throw "Compose render failed with rc=$LASTEXITCODE" }

On POSIX, replace NUL with /dev/null. Do not use /dev/null on Windows: Docker Compose resolves it as a Windows path and may read an unrelated C:\dev\null file instead of an empty env file.

The rendered configuration must show:

Repository verification

These commands use the MTP-aware targeted wrapper; confirm the reported ran-versus-suite count rather than relying only on the process exit code:

pwsh ./tools/scripts/test-targeted-xunit.ps1 `
  -Project src/JobEngine/StellaOps.Scheduler.__Tests/StellaOps.Scheduler.WebService.Tests/StellaOps.Scheduler.WebService.Tests.csproj `
  -Method '*EstateDoctorPluginStagingContractTests*'

pwsh ./tools/scripts/test-targeted-xunit.ps1 `
  -Project src/JobEngine/StellaOps.JobEngine.__Tests/StellaOps.JobEngine.EstateWorker.Tests/StellaOps.JobEngine.EstateWorker.Tests.csproj `
  -Method '*EstateDoctorRuntimePluginLoaderTests*'

pwsh ./tools/scripts/test-targeted-xunit.ps1 `
  -Project src/JobEngine/StellaOps.Scheduler.__Tests/StellaOps.Scheduler.WebService.Tests/StellaOps.Scheduler.WebService.Tests.csproj `
  -Method '*EstateDoctorBundle*'

The first check pins the one-Scheduler/five-Doctor packaging selections, recursive per-DLL signing, opt-in read-only mounts, fail-closed root setting, and absence from the default chain. The second pins exact-family membership, real family/check activation, digest/tamper rejection, per-DLL signatures, and real staged RSA admission. The third performs real RSA admission and one-byte tamper rejection for the Scheduler producer.

At the staged implementation rehearsal, the reported counts were 2/2 for the packager/overlay contract, 9/9 with zero skipped for the worker loader, and 2/2 for Scheduler signed admission/tamper. Without staged artifacts the loader’s artifact-conditioned method explicitly skips; that result is not staging evidence. Any other ran-versus-suite count is a stop, even when the command exits successfully.

Window handoff and stop

Stop after staging and repository verification. Before any up, recreate, or live forcing call, the owner-attended window must separately confirm the signed bundle/trust-root inputs, JobEngine connection, worker role and tenant, candidate images, service-scoped rollback, and the intended positive and negative forcing calls. Starting the profile is a deployment and is outside this runbook’s repository-side authorization.

Estate activation record (2026-09-10)

Executed on the lab estate under the owner-directed SPRINT_20260722_009 DOC-4b window. Everything below is the additive subset of the JobEngine cutover (SPRINT_20260722_012 JOB-9): a database, a role, environment keys, the opt-in overlay, and two service recreates. The data move, grant revocation, soak and gateway route swap are NOT part of it and remain JOB-9’s.

Preconditions that were measured, not assumed

Steps, in the order that worked

  1. Database + role (idempotent): PGPASSWORD_SERVICE="$JOBENGINE_POSTGRES_PASSWORD" bash tools/scripts/deploy/postgres/provision-service-database.sh stellaops_jobengine jobengine — jobengine is NOSUPERUSER NOBYPASSRLS, PUBLIC connect revoked.

  2. devops/compose/.env: JOBENGINE_POSTGRES_USER, JOBENGINE_POSTGRES_PASSWORD, STELLAOPS_POSTGRES_JOBENGINE_CONNECTION (ENV-001: the Password= literal must match), JOBENGINE_IMAGE_TAG, ESTATE_DOCTOR_WORKER_ROLE=docker-readonly, ESTATE_DOCTOR_TENANT=default. All six are documented in devops/compose/env/stellaops.env.example.

  3. Bundles: the two packager commands from Stage the signed bundles (lab: -UseOfflineDevSigner), then the inspection block. Expect one producer bundle with no StellaOps.Doctor*.dll, five Doctor family bundles, every managed DLL signed (63 of 63 on 2026-09-10), cosign.pub present under both trust-root directories.

  4. Images: IMAGE_TAG=dev bash devops/docker/build-service-publish.sh jobengine-web jobengine-estate-worker from a clean committed checkout; read /app/buildinfo.json inside each image (MSYS_NO_PATHCONV=1 docker run --rm --entrypoint cat <image> /app/buildinfo.json) and expect worktreeState=clean. Pin both by image ID in docker-compose.local-jobengine-web.yml and docker-compose.local-jobengine-estate-worker.yml (deploy by digest, never by tag).

  5. Render and parity-diff before recreating: render the chain below with config, extract the jobengine-web service, and diff its environment and mount targets against docker inspect of the running container. On 2026-09-10 the only deltas were the estate keys, the two new read-only mounts, and Router__AuthorizationTrustMode=GatewayEnforced (folded into the per-service pin because the tracked estate-wide setter is not in this chain).

  6. Recreate the estate worker first (it converges the schema into the empty database as the owner role), then seed, then recreate jobengine-web:

    cd devops/compose
docker compose -p stellaops --project-directory "$PWD" \
  -f docker-compose.stella-ops.yml \
  -f docker-compose.jobengine.yml \
  -f docker-compose.plugins.scheduler-estate-doctor.yml \
  -f docker-compose.local-jobengine-web.yml \
  -f docker-compose.local-jobengine-estate-worker.yml \
  -f docker-compose.existing-networks.override.yml \
  --profile estate-doctor \
  up -d --no-deps --force-recreate jobengine-estate-worker

    Expect in its log: Applied 7 migration(s) for JobEngine.Consolidated, Applied 4 migration(s) for Eventing.Reliability, and polling the estate-doctor queue for 1 tenant(s) as role 'docker-readonly' granting [docker.socket.read]. Never --remove-orphans.

  7. Seed stellaops_jobengine(as the superuser; the tables force RLS and the superuser bypasses it): one packs.packs row — pack_id = '<manifest.packId>@<manifest.version>'(the canonical registry identity, D-JEF-3; name = the manifest’s packId, version = its version, so name || '@' || version = pack_id), status='published', content = the exact bytes of the committed manifest, digest = 'sha256:<hex of content>' (verify in SQL: encode(sha256(content),'hex') = substr(digest,8)) — and one scheduler.schedules row with job_kind='estate-doctor', plugin_config = {"packId": "<name>@<version>", "actionIds": [...]} (the plugin refuses a bare packId at validation) and a non-NULL estate_constraints document (NULL means CLOSED). Note: on jobengine-web’s first start the JOB-11 storage reconciler moves the inline bytes to the upload store (compose_packsregistry-upload-data) and clears content; both estate readers follow that move (F-B, SPRINT_20260914_003 JEF-3), and the estate worker mounts the same volume read-only. The benign pack used is docs/modules/doctor/estate-pack-scheduled-diagnostics-manifest.json (sha256 81f4ee2c9e24003d207c8f60d3cea206da084f366c0ba1ea91824239c4c21480): five read-only Docker checks, requiresApproval=false — cron carries no approval token, so an approval-requiring action is refused ApprovalRequired by construction and can never be the scheduled proof. Every created_by/updated_by/deleted_by value must be a simple identifier (letters, digits, -, _, ., /): the model validates them on read, and one non-conforming ENABLED row makes the whole tenant’s cron pass throw until it is repaired (measured: 16 failed passes from one hand-written updated_by with a colon and spaces).

  8. Recreate jobengine-webon the same chain (same command, service jobengine-web). Expect the Router registration and Cron triggering enabled in its log; /app/plugins/scheduler/estate-doctor and /app/trust-roots/plugins/scheduler/cosign.pub mounted inside the container.

  9. Observe. Producer (jobengine-web): Cron trigger created run run-cron-<schedule>-<epoch> … kind=estate-doctor then estate-doctor dispatched N of N action(s) from pack '<packId>'; platform scheduler.runs row completed. Worker: Estate run <id> (tenant default) finished in …: Action '<checkId>' completed 1 check(s). Records in stellaops_jobengine: scheduler.estate_run_dispatches (claimed), estate_run_queue (outcome='Executed', attempts=1), estate_approval_audit (decision='Allowed', approval_token_present=false), estate_dispatch_ledger. Prove the tenant scoping with the real role: PGOPTIONS="-c app.tenant_id=default -c app.current_tenant=default" psql -U jobengine … sees the rows, another tenant sees none, no tenant context is refused (P0001). Then bash tools/scripts/deploy/postgres/probe-database-isolation.sh (append the vulnerability hub’s live connection first, per the script’s own trap note) — expect PASS with the jobengine pair in the matrix.

  10. Settle the cadence. The proof ran on */2 * * * *; steady state is 0 * * * * in the schedule row with maxExecutionsPerWindow=5 per action per hour.

What the first live run found

The first two occurrences failed with P0001: app.tenant_id session variable not set, and the worker’s second poll onward failed the same way while its first poll had succeeded. TenantScopedDataSources set the RLS GUC once per physical connection; Npgsql’s DISCARD ALL on pool return runs RESET ALL, which clears a set_config value but RESTORES a startup-packet value. Every prior suite and the 2026-08-17 rehearsal had run as a superuser, which bypasses FORCE ROW LEVEL SECURITY, so nothing could observe it. Fix 75211c3711: the tenant rides in the connection’s startup Options (-c app.tenant_id=… -c app.current_tenant=…) after validation as a plain token; TenantScopedDataSourcesTests pins it under a NOSUPERUSER NOBYPASSRLS owner role. Rebuild the worker image and RE-PACKAGE the producer bundle after any change to StellaOps.JobEngine.Persistence, and jobengine-web needs a recreate to reload it. Since 2026-09-14 (SPRINT_20260914_003 JEF-3, D-JEF-8) the packager prunes every host-shared assembly from the producer bundle before it is signed, so the bundle ships only StellaOps.Scheduler.Plugin.EstateDoctor.dll: the consolidated jobengine-web host carries all 17 former dependencies in /app, and a bundle copy of any of them loads into the plugin’s own load context, where the IEstateScheduleConstraintsSource the plugin’s stores implement is not the host’s type — every cron fire then fails Entry point was not found, whatever the bundle’s build stamp says. A bundle that lists more than its own assembly is wrong even if freshly packaged.

Rollback (each switch independent)