Estate doctor packs + self-healing rehoming — design (DOC-4b)

Status: LIVE on the lab estate since 2026-09-10 — estate packs run on schedule through the non-superuser jobengine role with results in run records; SPRINT_20260722_009 DOC-4b is DONE and archived. The activation record and the operating procedure are in estate-doctor-plugin-staging.md. Sprint: SPRINT_20260722_009 DOC-4b. Coordinates with: SPRINT_20260722_012 (JobEngine consolidation), ADR-026 (doctor integration coverage), ADR-030 (signing roles), ADR-039 D11/P11, DC-14.

Verified against: runtime adoption 8f1ce2a41d74732b1a900089aa1ee22143d9210b, held-back packaging/overlay c8efe1397e80161f5abb625b159aceac894e6b62, integration-runtime/Core split 1fa385c0b53b70dceefd232f72ff4df245270e76, and exact-family admission ab32c14569547aaca53de9c69366d0756fb07657. Re-verify with pwsh tools/scripts/build-boundary/generate-build-boundary-report.ps1 -Check, the targeted EstateWorkerClosureConformanceTests, EstateDoctorRuntimePluginLoaderTests, EstateDoctorPluginStagingContractTests, and SchedulerPluginAdmissionTests estate-doctor methods.

Companion to plugin-hosting-design.md, which designed the per-service doctor-check/v1 SDK and explicitly deferred everything here: “the old interface remains for the estate-level packs until DOC-4b rehomes them” and “ADR-026 heal declarations map onto JobEngine Task Packs in DOC-4b”.

0. Execution settlement (2026-08-25)

The repo-side path now exists without compiling a Doctor implementation into JobEngine:

• the Optional SchedulerEstateDoctorJobPlugin has an estate-doctor package descriptor and a held-back Compose overlay; the default include chain remains unchanged;
• the privileged worker admits exactly the Docker, Observability, ServiceGraph, Security, and Core.Worker doctor:checks bundles from the estate-doctor signed runtime-bundle profile, rejects missing or unexpected family directories, verifies the primary manifest/digest and every managed DLL’s detached signature before activation, and retains zero build-boundary violations;
• the temporary one-check worker substitute is retired. The pinned benign pack now targets the real Docker family action check.docker.socket; required-worker mode fails when the socket is absent and completes an HTTP /_ping before it can pass.

(Superseded 2026-09-10.) This was repository-side staging, not a live schedule forcing: at the time no database was created, no profile was enabled, no bundle was published, and no socket-mounted estate worker was recreated. On 2026-09-10 the live schedule forcing happened on the lab estate through the additive subset of the JobEngine window (database + role + held-back overlay/profile): two cron-fired runs each dispatched 5 of 5 actions of the benign estate.scheduled-diagnostics pack and the privileged worker executed all ten as the non-superuser role, with the run records tenant-scoped by the row-level-security policies. The first live run as that role exposed and fixed a defect in the tenant-pinned data source (the GUC did not survive Npgsql’s pool reset; it now rides in the startup packet — 75211c3711). Bundles remain lab-signed; production signing/publication is separate. Q-20 is implemented: Observability, ServiceGraph, Security, and the nine-check Core.Worker subset now execute through the verified-closed runtime seam and Core split described in §6; Core’s three host/loader introspection checks deliberately remain host-side. A post-change ephemeral-key rehearsal rebuilt, signed, re-hashed, and admitted the exact five-family profile without publishing or activating it.

1. Measured baseline (2026-08-10) — read this before planning any of it

Every line below was verified against src/, because DOC-4b’s carry-list was written in 2026-07-23 and two of its premises have since become false.

What exists in JobEngine today

• SchedulerDoctorJobPlugin (src/JobEngine/StellaOps.JobEngine.__Libraries/StellaOps.JobEngine.Scheduling/DoctorJobs/SchedulerDoctorJobPlugin.cs) is registered unconditionally (Scheduler.WebService/Program.cs:284-287), job kind doctor, enabled in the job-kind catalog (SchedulerJobKindCatalog.cs:29).
• It is an HTTP driver of doctor-web, not an in-process runner. POST /api/v1/doctor/run → poll GET /api/v1/doctor/run/{id} → GET …/results (:529-744), against DoctorJobOptions.DoctorApiUrl, whose default is http://doctor.stella-ops.local (:995).
• Three system schedules are seeded on every startup for tenant default (Bootstrap/SystemScheduleBootstrap.cs:25-43): doctor-quick-hourly (0 * * * *), doctor-full-daily (0 4 * * *), doctor-compliance-weekly (0 5 * * 0).
• Results land in scheduler.doctor_trends via PostgresDoctorTrendRepository, exposed by four read-only endpoints under /api/v1/scheduler/doctor/trends*.

What does NOT exist (each verified by exhaustive grep, not by absence of memory)

• No approval machinery of any kind in JobEngine. No approval table, type, endpoint, or token. A case-insensitive approval|approve sweep over every .cs in src/JobEngine/ returns exactly one hit, and it is a test string literal.
• No maintenance-window or schedule-constraint type. scheduler.schedules has cron_expression, timezone, only_if, limits — no window, no blackout.
• No pack execution path. PacksRegistry stores, signs, mirrors and lifecycles packs; it never runs one. No pack job kind exists among the 17 in the catalog.
• No estate-doctor queue, no jobengine-estate-worker, no privileged worker container.
• No docker socket is mounted anywhere except agent-core(docker-compose.stella-services.yml:2234). doctor-web does not have it either, so the entire Docker check family is privilege-starved in the live stack today.
• packs.pack_versions does not exist. It was deliberately dropped in the JobEngine consolidated baseline, with the rationale that it never had a reader or a writer.

Two premises that are now false, and what replaces them

1. DC-14’s grounding. The register says the capability allowlist is a pack_versions.manifest_json field. That table is gone (JobEngine’s D-JOB4-2 flagged this and declined to carry dead columns to make the register look grounded). Re-grounded in §4.
2. “Doctor.Remediation is the gated self-healing engine we port 1:1.” It is gated shut. RemediationOptions is never bound to configuration anywhere in the repo — the only two registrations are bare services.AddOptions<RemediationOptions>(), there is no Doctor:Remediation section in any appsettings or compose file, so Enabled is always false and every call short-circuits at RemediationGate.cs:63-69 with NotEnabled and returns manual guidance. The endpoint, the scope (doctor:remediate), the durable Postgres audit sink and the doctor.remediation_audit table are all real and wired; the executor has never executed. See §5 for why this changes the mapping’s risk profile rather than its content.

2. Decision D-4b-1 — scheduler.doctor_trends retires; it is not rehomed

This closes JobEngine’s D-JOB3-6, which has been blocked on 009 since 2026-08-03.

Measured: the table is a write-only store with four unused read endpoints. The only writer is SchedulerDoctorJobPlugin. Of the readers — the four /trends* endpoints — the sole caller in the entire repo is SchedulerDoctorTrendEndpointTests. The console has a typed client method (HttpDoctorClient.getTrends) that no component, store, resolver, guard or effect invokes; the console’s own audit evidence records the last consumer being removed (_evidence/20260717_manual_review/fable-audit-operate.md:120: “0 calls observed (was 2×). Gone.”). The CLI has zero references. No runbook instructs anyone to call it.

Decision. The table, its four endpoints, and the console’s orphaned getTrends path retire together with the HTTP-driven doctor job kind. They are not migrated into JobEngine run records and not carried forward as a bespoke trend table.

Why not “rehome into run records” as the DOC-4b task text originally said. Rehoming means preserving a shape someone reads. Nobody reads this one. Copying a write-only store into a new home buys a second write-only store plus a migration. Trend history, when it is genuinely wanted, is a derived view over estate-pack run records — JobEngine already persists scheduler.runs + run_summaries with declared retention classes — and it should be built when a consumer exists, not in advance of one.

Honesty cost, stated. Retiring it means the platform has no check-level trend series for a while. That is the current situation in every way that matters (nothing renders one today); what changes is that the store stops pretending otherwise. The registry doctor source already returns an honestly empty history for the same reason, and says so in registry-doctor.client.ts:126-131.

Sequencing. The table drop is destructive and follows the same rule as the doctor schema (DOC-5 stage 5): recorded owner approval, after the successor lands. Deleting the endpoints and the dead console method is severable earlier, with the job kind.

3. The DOC-5 sequencing hazard this design must own

SchedulerDoctorJobPlugin is an HTTP client of doctor-web, and the DOC-5 stage-4 inventory does not list it.

The runbook correctly notes that the gateway route ^/api/v1/scheduler/doctor(.*) stays because it is JobEngine’s, not doctor-web’s. That is true of the route and misses the job behind it: stage-4 item 5 deletes the doctor-web compose key, at which point all three seeded schedules begin failing — doctor-quick-hourly every hour — against a host that no longer resolves. The schedules are seeded unconditionally at startup, so an operator cannot avoid this by not creating them.

Consequence for the window: the doctor-web deletion set is not complete without one of

• (a) removing the doctor job kind and its three seeded schedules in the same change-set, or
• (b) landing DOC-4b’s estate packs first so the job kind has a non-HTTP implementation.

EXECUTED 2026-08-17 (DOC-5 stage 4): option (a). The doctor job kind, its three seeded schedules, SchedulerDoctorJobPlugin and PostgresDoctorTrendRepository were retired in the SAME change-set as the doctor-web compose key, exactly as this section required. The scheduler.doctor_trends TABLE was not dropped — that is stage 5, destructive and owner-gated.

One correction this execution forced, recorded here because §3 is where the next reader will look: the runbook’s item 6 note that ^/api/v1/scheduler/doctor(.*) “STAYS — it is JobEngine’s” is still true about OWNERSHIP but no longer true about service. Nothing in src/ serves that prefix once the job kind is gone. It was left in place anyway: it belongs to JOB-9’s atomic ten-route swap (api-path-prefixes.ts pins it as jobengineSchedulerDoctor), so removing it here would take one route out of another sprint’s coordinated window.

(a) is the smaller change and is the right move if DOC-4b is not ready when the flip soaks; it is recorded in the sprint as a stage-4 addition rather than left to be discovered by an hourly alarm. This is a checklist gap, not a flip blocker: stage 1 (the config flip) does not touch doctor-web, so the hazard only arrives at stage 4.

4. DC-14 re-grounded — where the capability allowlist actually lives

DC-14’s resolution stands; only its mechanics need re-grounding, since packs.pack_versions is gone.

What the runtime actually persists is packs.packs, which carries digest, signature, provenance_uri, provenance_digest, content BYTEA, provenance BYTEA and status packs.pack_status, plus packs.attestations (DSSE, retention-class evidence), packs.lifecycles and packs.audit_log (retention-class audit).

Re-grounding. The capability allowlist is a field of the signed pack manifest inside packs.packs.content, covered by digest and therefore by signature — not a column. This is strictly stronger than the original framing: a column can be updated without invalidating a signature, whereas a manifest field cannot be altered without breaking the digest the attestation covers. Enforcement is two-sided and unchanged in spirit:

• At publish, PacksRegistry parses the manifest out of the verified content and rejects a pack declaring a capability its signing role (ADR-030) is not permitted to grant.
• At run, the estate worker intersects the pack’s declared capabilities with its own role’s allowed set and refuses the difference. The worker must re-derive capabilities from the signed content, never from a registry column, so a registry compromise cannot widen a pack’s powers.

If multi-version Task Packs later need a per-version catalog, that table is authored then, with a reader and a writer, per JobEngine’s D-JOB4-2 ruling.

A naming collision that must not become a security hole. DoctorPack already exists and is a different thing: src/__Libraries/StellaOps.Doctor/Packs/ is a declarative YAML check-pack format (DoctorPackLoader, search path Doctor:Packs:SearchPaths, default {root}/plugins/doctor) whose checks are shell commands (DoctorPackCommandRunner), with no signature verification, no approval and no capability allowlist. It is wired into the engine today (CheckRegistry.cs:16,24).

Loading that format on a socket-mounted privileged worker would be arbitrary shell execution with root-equivalent power, reached without touching the signed-pack path at all. The estate worker must not load the YAML pack format. Estate packs are JobEngine Task Packs (signed, digest-attested); the YAML format stays with the CLI/engine or is retired with it. Use the term “estate pack” for the former and never for the latter.

Loader isolation (measured 2026-08-17). The loader has no enable/disable control. Measured against src/: DoctorPackLoader is a sealed class taken as a required, non-virtual constructor dependency of CheckRegistry(CheckRegistry.cs:16,24), and both are TryAddSingleton-registered. There is no flag, no interface to null-object and no configuration key — any host that builds a CheckRegistry gets the shell-exec loader.

The isolation is therefore STRUCTURAL, not configured: the estate worker must not reference the Doctor engine assembly at all, and that is closure-verified rather than reviewed. §6’s contracts were extracted into StellaOps.Doctor.Contracts (a CoC §2.7A closed-contract project with zero StellaOps.* ProjectReferences) precisely so §4 and §6 can both hold; Packs/ stays in StellaOps.Doctor. The existing DoctorSdkAdoptionBoundaryConformanceTests.EveryAdoptingServiceKey_CompilesTheSdk_AndNeverTheCentralDoctorEngine already fails any deployable key whose closure reaches the engine, with adoption derived from the closure rather than listed.

5. ADR-026 gating mapped onto run approval

The port is 1:1 in semantics; the substrate changes from an in-process gate to a run-approval decision made before a run is dispatched to the estate queue.

These current JobEngine equivalents were verified against 773ddc7bbc996531ac4db789b1a5ee2b64f3018b. Re-verify them with rg -n "MaintenanceWindow|DurableRunApprovalRateLimiter|PostgresRunApprovalTokenStore" src/JobEngine/StellaOps.JobEngine.__Libraries.

Preserve the evaluation ORDER, not just the inputs. RemediationGate.cs:60-141 evaluates: enabled → action-enabled → severity threshold → (if destructive or approval-required) token present, dry-run command present, remediate scope, durable audit → maintenance window → rate limit. The destructive branch is the load-bearing part: a destructive action with no DryRunCommand is refused even when everything else passes.

One deliberate cleanup to make during the port. RemediationDecisionKind declares ApprovalRequired, DryRunPreviewMissing, ScopeMissing and AuditUnavailable, and the gate produces none of them — all four collapse into ManualFallback. The audit row therefore records that an action was refused but not which precondition refused it. The run-approval port should emit the specific kinds, since a refusal reason is exactly what an operator reads at 3am. This is a fidelity improvement over 1:1 and should be called out as an intentional divergence.

Risk note. Because the current gate has never allowed anything (§1), the port has no production behaviour to regress — and equally, no production evidence that the gate’s allow-path is correct. The first estate pack with a destructive action must therefore ship with the gate’s allow-path exercised in tests, not merely inherited.

6. Estate packs — what moves and what it needs

Five families run on the privileged worker: Docker (5 checks), Observability (7), ServiceGraph (11), Security (15), and Core.Worker (9), for an exact 47-check set. The other three original Core checks inspect the host’s plugin registry/loader and remain in the host-side Core family. The worker set keeps the existing IDoctorPlugin/IDoctorCheck contract — deliberately, because unlike per-service checks these genuinely need the estate view that DoctorPluginContext carries, which is precisely why doctor-check/v1 refused to carry it.

Contract home corrected 2026-08-17 (DOC-4b-C). Those three types now live in src/__Libraries/StellaOps.Doctor.Contracts/, not in the engine. When this design was written they sat in src/__Libraries/StellaOps.Doctor/Plugins/ — the same assembly as Engine/CheckRegistry.cs and Packs/DoctorPackLoader.cs — which made §6 and §4 contradictory: satisfying §6 by project reference dragged §4’s shell-execution hazard into the privileged worker permanently, and per §4’s correction there is no flag to switch it off once it is there.

The extraction moves the contracts plus their minimal closure (Plugins/Heal/*, DoctorCheckResultBuilder, the DoctorCheckResult model group, and the Detection.RuntimeEnvironment enum — RemediationCommand genuinely uses it). The later Q-20 implementation moved Integrations/ and Resilience/ into producer-owned StellaOps.Doctor.IntegrationChecks, whose exact closure is Contracts + Cryptography + transitive Localization. Packs/, Engine/, Output/, Export/, the IRuntimeDetector implementation, and DoctorPluginContextExtensions stay in the engine; the YAML loader and CheckRegistry therefore remain structurally absent from the privileged worker.

Namespaces did not change; only the containing assemblies did. StellaOps.Doctor carries [assembly: TypeForwardedTo(...)] shims for all 26 Contracts types and all 16 moved IntegrationChecks runtime types; host Core forwards the nine worker checks to Core.Worker. Dropping a forwarder is a silent runtime break for mounted bundles that compilation does not catch, because freshly-built consumers bind straight to the new assembly; see docs/runbooks/general/plugin-cross-alc-missingmethod.md. The post-change exact-family coherent rebuild/re-sign rehearsal passed with an ephemeral scratch key; production signing/publication is a separate outward-facing action.

Execution update: all five runtime bundles are packageable and admitted without a JobEngine compile edge into Doctor implementation. Q-20 IMPLEMENTED (2026-08-26): verified-closed-runtime-seam — producer-owned StellaOps.Doctor.IntegrationChecks carries the integration/resilience runtime surface; Core splits nine worker checks from three host/loader introspection checks; Contracts stays data+check contracts; AddDoctorEngine/CheckRegistry/ DoctorPackLoader widening remains rejected. Settlement receipt: SPRINT_20260722_009 Decisions & Risks; implementation commits are named in this document’s verification header.

Two constraints fall out of the baseline:

• Docker checks need a socket that the default chain does not grant. All five read Docker:Host defaulting to unix:///var/run/docker.sock. The held-back worker path sets required-socket mode: absence is Fail and a pass requires a Docker HTTP /_ping. Repo-side AF_UNIX forcing is green; the first run on the live estate is still new behaviour and must be treated as such.
• The coverage the flip trades away is check.postgres.schemas.present— the estate-wide “did every service’s migration ledger converge” probe, which per-service SDK checks structurally cannot replace (each sees only its own database). Four ratchet suites in Doctor.Plugins.Core.Tests are keyed on it. This check is the single strongest argument for landing estate packs promptly after the DOC-5 flip rather than leaving them open-ended.

7. Execution order and current state (2026-08-26)

1. DONE repo-side: estate-pack contract + estate-doctor job kind, queue, and producer plugin.
2. DONE repo-side / held back operationally: jobengine-estate-worker role container, queue-only subscription, capability intersection, structural exclusion of the YAML engine, and signed exact-five-family runtime-bundle admission. The socket/profile overlay is not in the default chain.
3. DONE: run-approval token, decision record, durable audit, maintenance window, rate limit, and schedule constraints (§5).
4. DONE repo-side: Docker, Observability, ServiceGraph, Security, and Core.Worker are admitted as an exact five-family/47-check set; missing or unexpected families and unsigned/tampered DLLs fail closed. The temporary substitute is retired. The HTTP-driven doctor job kind and its seeded schedules already retired with DOC-5 stage 4.
5. DONE by DOC-5 stage 5: doctor_reports, remediation_audit, doctor-evidence, and scheduler.doctor_trends were dispositioned under the recorded pre-release ruling.

Nothing is owed (2026-09-10). The schedule-forcing run happened on the lab estate (see the activation record in the staging runbook): the profile is active, stellaops_jobengine exists and holds the run records, and the plane runs hourly. No artifact was published outward; the bundles are signed with the repository’s offline dev signer, which is the lab posture.