ADR-026: Doctor integration-coverage clause, structural probe hardening, and gated self-healing

Status: Accepted (design-only) Date: 2026-06-08 Sprint: SPRINT_20260608_005_Doctor_integration_clause_contract.md (with 003, 004, 006, 007, 008) Related ADRs: ADR-004 (forward-only migrations — remediation-audit + manifest persistence), ADR-006 (EPF — heal actions dispatch capabilities like restart/redeploy), ADR-024 (silent-green anti-pattern — swallowed failure + conflated readiness)

Read this before adding a Doctor check, touching the Doctor engine/plugin loader, or wiring any remediation/execution path.

Context

Doctor is the operator’s “is the platform healthy” surface. A whole-repo audit on 2026-06-08 (21-agent sweep, cached at workflow wbivcw3to) inventoried 250 external/internal dependency points against 121 existing checks and found the engine bones are sound but coverage is broad-but-shallow, substantially dead, and “healing” is almost entirely guidance rather than execution. The failure shape mirrors ADR-024: a green light that proves nothing.

Verified facts (file:line):

  1. Engine isolation already exists — but with one hole. CheckExecutor.ExecuteCheckAsync wraps every check in try/catch and converts a throw into a Fail result, so one bad check never aborts the run (src/__Libraries/StellaOps.Doctor/Engine/CheckExecutor.cs:91-111). However check.CanRun(context) is invoked outside that try (CheckExecutor.cs:85) — a throwing CanRun crashes the parallel body. The per-check timeout is cooperative and a single global 30s (DoctorRunOptions.Timeout; EstimatedDuration is collected but never used to size it), so a check doing blocking I/O that ignores its token stalls a worker for the full hang. Plugin discovery (CheckRegistry.GetChecks/IsAvailable) runs untimed.
  2. Coverage looks greener than it is — config-key drift. Several real checks read the wrong configuration key and therefore silently Skip on the actual deployment: Valkey reads Valkey:ConnectionString while the platform uses Router__Messaging__valkey__ConnectionString/ConnectionStrings__Redis (ValkeyConnectivityCheck.cs:37-44); the gateway entry in ServiceEndpointsCheck gates on StellaOps:GatewayUrl vs the real Router__Gateways__0__Host (ServiceEndpointsCheck.cs:64); the Database check reads ConnectionStrings:DefaultConnection vs ConnectionStrings__StellaOps (DatabaseCheckBase.cs:109-120). A skipped check reports nothing and looks healthy.
  3. Dead and forked plugins inflate the count. Registered in no host: Plugins.Authority (AddAuthorityDoctorPlugin uncalled), Plugins.Cryptography (AddDoctorCryptographyPlugin uncalled), in-process Plugins.Notify (no DI extension), mounted Plugin.Notify (absent from DoctorRuntimePluginTargets), Plugin.Policy (no csproj), both Agent variants, AI, Sources. Three capabilities exist as duplicate forks (in-process StellaOps.Doctor.Plugins.* vs mounted StellaOps.Doctor.Plugin.*) — Crypto, Notify, Postgres/Database — with divergent code and the same bugs.
  4. Fake-green stubs. Checks that return hardcoded data while looking like probes: the entire Operations plugin; 3 of 4 Auth checks (SigningKeyHealthCheck returns the literal "key-2024-01-15"); the 6 Crypto checks; all 3 Vex checks; the Attestor cosign Fulcio/KMS/Vault paths; PostgresMigrationStatusCheck.GetPendingMigrationsAsync (always empty); EvidenceLocker Merkle verification (field-presence only). check.postgres.migrations probes the wrong table (public.__EFMigrationsHistory, EF) vs the real <schema>.schema_migrations (MigrationRunner.cs:419).
  5. Checks can vanish and Doctor still reports all-green. Mounted plugins load via Assembly.LoadFrom into the default ALC + GetTypes() (DoctorMountedRuntimePluginLoader.cs:476,549-556); a missing trust-root (cosign.pub) or unresolved transitive dep makes the loader silently Reject the whole plugin (:341-343), and its checks disappear from the run. /readyz returns static Ok (Program.cs:219) regardless. Nothing probes loader health.
  6. “Healing” is guidance, not execution. The engine has no executable remediation abstraction: IDoctorCheck.RunAsync only yields a DoctorCheckResult carrying a descriptive Remediation record of command strings (Models/RemediationStep.cs). The only generic executor is the CLI stella doctor fix --apply, hard-gated to stella-prefixed shell commands (DoctorCommandGroup.cs:895-958) — it cannot run systemctl/psql/docker. Timestamping’s IAutoRemediationService is the only executable auto-heal framework, but it is plugin-local, wired to no-op Unavailable* stubs, and never invoked by any host (AutoRemediation.cs; NullDependencyProviders.cs:92-139). The UI “Run Fix” button copies to clipboard. A mature, reusable remediation engine already exists at src/ReleaseOrchestrator/__Libraries/StellaOps.ReleaseOrchestrator.Environment/Inventory/Remediation/RemediationEngine.cs (plan/execute split, severity-gating, rate-limit, evidence writer, maintenance-window guard) and is not referenced by Doctor.

The platform is offline / air-gap-first and supply-chain-sensitive. The fix must make coverage and healing structural invariants (impossible to skip, impossible to silently lose) rather than per-author discipline, and any execution must be locally-gated with durable evidence.

Decision

  1. One integration manifest is the source of truth for “all integrations.” Introduce a declared IntegrationCatalog — every external and internal-3rd-party dependency (Postgres per schema, Valkey, gateway/router, identity-envelope, each sibling service, TSA, Rekor, Vault, OCI/GitLab registries, SmRemote/HSM, NATS, OTLP, SMTP, OCSP/CRL/LOTL, debuginfod, object storage, …) is a registered entry carrying its id, kind, canonical config keys, blast radius, and owning check id. A meta-check check.core.integration-coverage fails when any manifest entry has no live check. Adding an integration without a check is now a red Doctor run, not an oversight.

  2. IntegrationCheck base class makes try/catch structural. Integration checks derive from a base that (a) runs the author’s probe body inside the canonical typed handler (CheckExceptionLogging.IsTransientProbeException/IsParseException → Fail-with-remediation; unexpected types still bubble to the engine), (b) resolves config keys from the manifest (eliminating drift — a check can no longer read a stale ad-hoc key), and © forbids swallow-to-Pass: a probe that cannot reach its dependency must emit Fail/Warn/Skip(reason), never Pass. “Each integration tries/catches against failure” becomes a property of the base, not a thing authors remember.

  3. Every integration check must offer a heal hook. The check contract gains heal actions (IReadOnlyList<IRemediationAction> GetHealActions(DoctorCheckResult) / Task<RemediationOutcome> HealAsync(...)). Each IntegrationCheck must return at least one IRemediationAction or explicitly declare ManualOnly(reason); the BareCatchAnalyzer companion rule and a registration test enforce it. IRemediationAction is a runnable, idempotent operation (Id/Description/IsDestructive/RequiresApproval/DryRun), not a command string.

  4. Executable remediation moves to the engine and reuses the existing engine. Lift the IRemediationAction/runner pattern out of Timestamping into the shared Doctor engine, backed by the ReleaseOrchestrator.Environment RemediationEngine (plan/execute, SeverityScorer gating, RemediationRateLimiter, IRemediationEvidenceWriter, maintenance-window guard). Surface it as POST /api/v1/doctor/remediate (new doctor:remediate scope, audited), a CLI verb, and real UI wiring. Audit is durable (Postgres, forward-only per ADR-004) — replacing the volatile/no-op stubs.

  5. Self-healing posture: full gated auto-heal — this supersedes the src/Doctor/AGENTS.md “destructive = manual guidance only” rule. Operators chose maximum automation with maximum safety:

    • Auto-execution is OFF by default; enabled per-action via config, never globally implicitly.
    • Non-destructive heal actions (refresh, restart-to-converge, create-dir, failover, cache-warm) may auto-run within rate-limit + maintenance-window once enabled.
    • Destructive heal actions are now permitted to auto-execute — but only behind the full gate: explicit per-run approval token, a dry-run preview (DryRunVariant), an active maintenance-window, durable audit, rate-limit, and the doctor:remediate scope. Absent any gate element, a destructive action falls back to manual guidance.
    • Air-gap invariant: all execution is local; no cloud-managed services, no external callbacks, no telemetry egress. Heal actions that would mutate state outside the host (e.g. external registry writes) are non-destructive-only and respect existing opt-in flags. The module AGENTS.md “Evidence and remediation” section is updated to reference this ADR as the governing policy.

  6. Plugin-load and readiness honesty closes the “checks vanish → all green” hole. Add check.core.plugins.loaded (expected-vs-actually-admitted plugins, surfacing DoctorPluginProbeReporter rejection reasons) and check.core.plugins.trustroot (trust-root presence/validity). /readyz fails when mounted-plugin admission was rejected, the trust-root is missing, the report-store is unreachable, or the identity-envelope key is absent. A hollow-but-“ready” Doctor is no longer possible.

  7. Truth-up is a precondition, and fake-green is forbidden. Dead plugins are removed or wired; exactly one implementation tree is chosen per capability; and a check may never return a hardcoded Pass— a not-yet-implemented probe must Skip(reason) (visible as “not verified”), never simulate success. Stubbed checks are either implemented as real probes or converted to honest Skip.

  8. Engine resilience hardening. Move CanRun inside the executor try; add a discovery-time timeout for GetChecks/IsAvailable; add a watchdog so a blocking check that ignores its token is reported (not silently slot-hogging); size the per-check timeout from EstimatedDuration; add a bounded retry/jitter policy distinct from the global timeout so a one-off blip is not a false alarm; and escalate DOCTOR0001 to error severity with extended scope (swallow-to-Pass and the non-IDoctorCheck helper classes checks delegate to).

Consequences