Doctor — Architecture

Source ownership, 2026-10-03. PCD-253 withdrew the uncompiled AdvisoryAI drafts under src/Doctor/__Libraries/StellaOps.Doctor/ and the old IDoctorPlugin.cs contract under src/Doctor/__Plugins/. The maintained AdvisoryAI implementation remains in src/__Libraries/StellaOps.Doctor/AdvisoryAI/; current plugin abstractions and checks keep their existing projects. See the per-file source disposition.

Verified against commit 942a18727c (2026-09-26, DOC-2 of SPRINT_20260926_001). Re-verify by reading the evidence paths listed in the Evidence section.

Audience: operators and SREs diagnosing a Stella Ops deployment, and engineers writing checks.

Doctor is the diagnostic framework: check contracts, the check engine and its plugins, the per-service doctor-check/v1 SDK, and the stella doctor CLI. There is no Doctor deployable. The central doctor-web host was retired on 2026-08-17 (SPRINT_20260722_009 DOC-5 stage 4) together with its compose keys, gateway routes, image publish key, host alias and the scheduler doctor job kind. The description of that host (HTTP API, scopes, report storage, mounted plugin loader, configuration) is kept as a frozen record in docs-archive/modules/doctor/architecture.md.

1) Where Doctor runs today

SurfaceWhat runsSource
Per-service checksEach adopting service hosts its own doctor-check/v1 checks at GET /doctor/<service>/checks, guarded by ops.health.src/__Libraries/StellaOps.Doctor.Plugin.Abstractions/ (AddServiceDoctorChecks, MapServiceDoctorEndpoints)
Registry and aggregatePlatform records which checks each service claims and serves the combined view the Console reads (GET /api/v1/platform/doctor/aggregate).src/Platform/StellaOps.Platform.WebService/Endpoints/DoctorCapabilityEndpoints.cs; see Platform architecture
Estate packsEstate-level diagnostics run as signed packs scheduled by JobEngine; the estate worker compiles only StellaOps.Doctor.Contracts, never the engine.src/JobEngine/StellaOps.JobEngine.EstateWorker/; design in estate-pack-design.md
CLIstella doctor runs the engine in-process with the Core, Database and BinaryAnalysis plugins.src/Cli/StellaOps.Cli/Program.cs, src/Cli/StellaOps.Cli/Commands/DoctorCommandGroup.cs

Per-service SDK (doctor-check/v1)

Release estate owner adoption (CCP-071)

check.release.environment.matches-approved is served by the existing Release Orchestrator IServiceDoctorCheck host, using its own repositories and the Watch classification. Its operational output contains owner-wide counts only. Unbaselined state is unhealthy/Critical; absent, failed, unknown, unevaluated or empty data is never healthy. The old Release plugin check and tests are frozen under src/__Obsoleted/Doctor/; the other four Release plugin checks remain. The neutral SDK is unchanged and no release family is added to the five-family estate-doctor worker profile. See the check article for the owner pull, the Platform registration settings, the identity grant (Authority Startup migration 040) and the activation checks. Source adoption and host tests do not establish live activation.

Withdrawn WebService source

The WebService source was withdrawn to src/__Obsoleted/Doctor/StellaOps.Doctor.WebService with its tests on 2026-09-27 (PCD-73, CoC 15.4). It had no compose file, release-bundle service, gateway route or image publish key since DOC-5 stage 4, and must not be restored to repair a leftover container (src/Doctor/README.md). Its persistence library is also withdrawn and unsupported (PCD-252, 2026-10-03): no live project referenced its marker type or embedded migration resources. The files are frozen unchanged at src/__Obsoleted/Doctor/__Libraries/StellaOps.Doctor.Persistence/ under CoC 15.4, and the unadopted persistence exception is removed. DOC-5 stage 5 recorded the drop of the legacy doctor schema and scheduler.doctor_trends on 2026-08-24. Current per-service checks, the Platform aggregate, JobEngine estate packs and the CLI use their existing owners; none adopts this retired report/remediation store. Recheck the withdrawal with the architecture pack’s PersistenceAdoptionConformanceTests and ObsoletedIsolationConformanceTests. The standalone Doctor Scheduler is frozen under src/__Obsoleted/Doctor/StellaOps.Doctor.Scheduler/, and the Timestamping plugin under src/__Obsoleted/Doctor/__Plugins/ (timestamp health is the Evidence family’s doctor.evidence.timestamp.* checks).

Gated self-healing

Remediation execution follows ADR-026 Decision 5 and is implemented DB-free in src/__Libraries/StellaOps.Doctor.Remediation (RemediationGate, RemediationOptions). Nothing runs unless RemediationOptions.Enabled is set and the action id is listed in EnabledActionIds. Destructive actions additionally require an approval token, a dry-run preview, an active maintenance window, durable audit, the rate limit and the doctor:remediate scope; otherwise they fall back to manual guidance. The live entry point is stella doctor remediate (--action, --approval-token, --dry-run).

Air-gap awareness

DoctorPluginContextExtensions.IsAirgapped (src/__Libraries/StellaOps.Doctor/Plugins/DoctorPluginContextExtensions.cs) lets a check short-circuit external probes in sealed mode with the canonical skip reason air-gap mode: external probe disabled; IsLoopbackOrPrivateAddress keeps internal collectors probeable.

2) Plugin Architecture

The interfaces below are the canonical contracts in namespaces StellaOps.Doctor.Plugins and StellaOps.Doctor.Models, compiled in src/__Libraries/StellaOps.Doctor.Contracts (Plugins/IDoctorPlugin.cs, Plugins/IDoctorCheck.cs, Plugins/DoctorPluginContext.cs, Models/DoctorCheckResult.cs); StellaOps.Doctor forwards the types. A separate, simpler IDoctorPlugin (Name/Categories/RunChecksAsync) exists in src/Doctor/__Plugins/StellaOps.Doctor.Plugins.Core for legacy aggregation; the engine and all built-in plugins use the contracts shown here.

Core Interfaces

public interface IDoctorPlugin
{
 string PluginId { get; } // e.g. "stellaops.doctor.integration"
 string DisplayName { get; }
 DoctorCategory Category { get; } // enum, not string
 Version Version { get; }
 Version MinEngineVersion { get; }

 bool IsAvailable(IServiceProvider services);
 IReadOnlyList<IDoctorCheck> GetChecks(DoctorPluginContext context);
 Task InitializeAsync(DoctorPluginContext context, CancellationToken ct);
}

public interface IDoctorCheck
{
 string CheckId { get; } // e.g. "check.integration.oci.referrers"
 string Name { get; }
 string Description { get; }
 DoctorSeverity DefaultSeverity { get; }
 IReadOnlyList<string> Tags { get; }
 TimeSpan EstimatedDuration { get; }

 bool CanRun(DoctorPluginContext context);
 Task<DoctorCheckResult> RunAsync(DoctorPluginContext context, CancellationToken ct);
}

DoctorCategory is an enum (Core, Database, Security, Cryptography, Attestation, Docker, Integration, Observability, Notify, ServiceGraph, Sources, Verification, Authority, AI, Environment, Release, Scanner, Infrastructure).

Plugin Context

public sealed class DoctorPluginContext
{
 public required IServiceProvider Services { get; init; }
 public required IConfiguration Configuration { get; init; }
 public required TimeProvider TimeProvider { get; init; }
 public required ILogger Logger { get; init; }
 public required string EnvironmentName { get; init; }
 public string? TenantId { get; init; }
 public IConfiguration PluginConfig { get; init; } // bound to Doctor:Plugins

 // Factory for building results, and a static credential redactor.
 public DoctorCheckResultBuilder CreateResult(string checkId, string pluginId, string category);
 public static string Redact(string? value);
}

Check Results

The result type returned by a check is DoctorCheckResult (StellaOps.Doctor.Models). The simpler CheckResult record (CheckId/Severity/IsHealthy/Message/Metadata) is a separate dashboard projection produced by DoctorEngine.RunChecksAsync.

public sealed record DoctorCheckResult
{
 public required string CheckId { get; init; }
 public required string PluginId { get; init; }
 public required string Category { get; init; }
 public required DoctorSeverity Severity { get; init; }
 public required string Diagnosis { get; init; }
 public required Evidence Evidence { get; init; }
 public IReadOnlyList<string>? LikelyCauses { get; init; }
 public Remediation? Remediation { get; init; }
 public string? VerificationCommand { get; init; }
 public required TimeSpan Duration { get; init; }
 public required DateTimeOffset ExecutedAt { get; init; }
}

public enum DoctorSeverity
{
 Pass = 0, // Check passed successfully
 Info = 1, // Informational (not a problem)
 Warn = 2, // Warning condition that should be addressed
 Fail = 3, // Failure (needs attention)
 Skip = 4 // Skipped (not applicable in current context)
}

Evidence carries a Description, an IReadOnlyDictionary<string, string> Data, and an optional SensitiveKeys list. Remediation carries ordered Steps (each with a CommandType of Shell/Sql/Api/FileEdit/Manual), a SafetyNote, RequiresBackup, and an optional RunbookUrl.

Engine resilience (ADR-026 D8)

The executor (Engine/CheckExecutor.cs) and registry (Engine/CheckRegistry.cs) are hardened so a single misbehaving check or plugin can never crash, stall, or silently shrink a run. See ADR-026 decisions D8 (engine resilience) and D6 (rejected/lost checks must not read as all-green).

Bare-catch analyzer scope (DOCTOR0001, ADR-026 D2/D7)

The BareCatchAnalyzer (src/__Analyzers/StellaOps.Doctor.Analyzers/BareCatchAnalyzer.cs) flags unfiltered catch { } / catch (Exception) so probe failures cannot be silently swallowed (the canonical fix is the catch (Exception ex) when (CheckExceptionLogging.Is…Exception(ex)) pattern; see §4 Check Patterns and src/Doctor/AGENTS.md).

3) Plugin libraries

Check families are grouped in plugin libraries. In-process families live under src/__Libraries/StellaOps.Doctor.Plugins.* (Attestation, Authority, Core, Core.Worker, Cryptography, Database, Docker, Integration, Observability, Security, ServiceGraph, Verification); further families live under src/Doctor/__Plugins/StellaOps.Doctor.Plugin.* (Agent, BinaryAnalysis, Compliance, Environment, EvidenceLocker, Notify, Operations, Release, Scanner, Storage). The integration catalog and its check bindings are in src/__Libraries/StellaOps.Doctor.IntegrationChecks.

No deployed host composes the full catalog: the CLI registers Core, Database and BinaryAnalysis, and only the WebService, withdrawn to src/__Obsoleted/Doctor/ by PCD-73, registered the remaining families. Check articles are required for every IDoctorCheck (docs/doctor/articles/<plugin>/<check-slug>.md, see src/Doctor/AGENTS.md). The OCI registry checks are described in registry-checks.md.

4) Check Patterns

Non-Destructive Probing

Registry checks use non-destructive operations:

// Pull check: HEAD request only (no data transfer)
var response = await client.SendAsync(new HttpRequestMessage(HttpMethod.Head, manifestUrl), ct);

// Push check: Start upload then immediately cancel
var uploadResponse = await client.PostAsync(uploadsUrl, null, ct);
if (uploadResponse.StatusCode == HttpStatusCode.Accepted)
{
 var location = uploadResponse.Headers.Location;
 await client.DeleteAsync(location, ct); // Cancel upload
}

Capability Detection

Registry capability probing sequence:

1. GET /v2/ → Extract OCI-Distribution-API-Version header
2. GET /v2/{repo}/referrers/{digest} → Check referrers API support
3. POST /v2/{repo}/blobs/uploads/ → Check chunked upload support
 └─ DELETE {location} → Cancel upload session
4. POST /v2/{repo}/blobs/uploads/?mount=...&from=... → Check cross-repo mount
5. OPTIONS /v2/{repo}/manifests/{ref} → Check delete support (Allow header)
6. OPTIONS /v2/{repo}/blobs/{digest} → Check blob delete support

Evidence Collection

All checks collect structured evidence via the fluent DoctorCheckResultBuilder created from the context. WithEvidence(description, configure) takes the evidence description plus a callback over EvidenceBuilder:

var result = context.CreateResult(CheckId, PluginId, Category)
.Pass("Registry authentication successful")
.WithEvidence("OCI registry authentication probe", eb => eb
.Add("registry_url", registryUrl)
.Add("auth_method", "bearer")
.Add("response_time_ms", (int)elapsed.TotalMilliseconds)
.AddSensitive("token_preview", DoctorPluginContext.Redact(token)))
.Build();

EvidenceBuilder.Add has overloads for string, bool, int, long, and double. AddConnectionString redacts the value automatically and marks the key sensitive; AddSensitive marks an already-redacted value sensitive.

Credential Redaction

Sensitive values are redacted by the static DoctorPluginContext.Redact helper (Plugins/DoctorPluginContext.cs). It shows the first 3 and last 3 characters for values longer than 8 characters; shorter or empty values collapse to a fixed token:

public static string Redact(string? value)
{
 if (string.IsNullOrWhiteSpace(value))
 return "[EMPTY]";
 if (value.Length <= 8)
 return "[REDACTED]";
 return $"{value[..3]}***{value[^3..]}";
}
// "mysecretpassword" → "mys***ord"
// "short" → "[REDACTED]"

5) CLI Integration

The stella doctor command group (src/Cli/StellaOps.Cli/Commands/DoctorCommandGroup.cs) runs the engine in-process. The CLI registers the Core, Database and BinaryAnalysis plugins (src/Cli/StellaOps.Cli/Program.cs). Subcommands: run (also the default action), list, export, fix and remediate.

stella doctor                                   # run all checks (default action == `doctor run`)
stella doctor --mode quick                      # quick | normal (default) | full
stella doctor --category Database --tag quick   # filter by category or tag (repeatable)
stella doctor --check <check-id>                # run one check
stella doctor --format json --output report.json
stella doctor --parallel 8 --timeout 60 --fail-on-warn
stella doctor --watch --interval 60             # continuous monitoring
stella doctor list --category Database
stella doctor export --output bundle.zip --include-logs --log-duration 4h
stella doctor fix --from report.json --apply    # dry-run unless --apply
stella doctor remediate --check <check-id> --action <action-id> --dry-run

Exit codes follow the run’s overall severity: failures map to CliExitCodes.DoctorFailed; warnings map to CliExitCodes.DoctorWarning only with --fail-on-warn.

6) Extensibility

Creating a Custom Check

public sealed class MyCustomCheck : IDoctorCheck
{
 public string CheckId => "check.custom.mycheck";
 public string Name => "My Custom Check";
 public string Description => "Validates custom integration";
 public DoctorSeverity DefaultSeverity => DoctorSeverity.Fail;
 public IReadOnlyList<string> Tags => ["custom", "integration"];
 public TimeSpan EstimatedDuration => TimeSpan.FromSeconds(5);

 public bool CanRun(DoctorPluginContext context)
 {
 // Return false if preconditions not met
 return context.Configuration["Custom:Enabled"] == "true";
 }

 public async Task<DoctorCheckResult> RunAsync(DoctorPluginContext context, CancellationToken ct)
 {
 var builder = context.CreateResult(CheckId, "custom", "Integration");

 try
 {
 // Perform check logic
 var result = await ValidateAsync(context, ct);

 if (result.Success)
 {
 return builder
.Pass("Custom validation successful")
.WithEvidence("Custom validation", eb => eb.Add("detail", result.Detail))
.Build();
 }

 return builder
.Fail("Custom validation failed")
.WithCauses("Configuration is invalid")
.WithRemediation(rb => rb
.AddManualStep(1, "Check configuration", "Verify Custom:Setting is correct")
.WithRunbookUrl("https://docs.stella-ops.org/runbooks/custom-check"))
.Build();
 }
 catch (Exception ex)
 {
 return builder
.Fail($"Check failed with error: {ex.Message}")
.WithEvidence("Check exception", eb => eb.Add("exception_type", ex.GetType().Name))
.Build();
 }
 }
}

Creating a Custom Plugin

public sealed class MyCustomPlugin : IDoctorPlugin
{
 public string PluginId => "custom";
 public string DisplayName => "Custom Checks";
 public DoctorCategory Category => DoctorCategory.Integration; // enum
 public Version Version => new(1, 0, 0);
 public Version MinEngineVersion => new(1, 0, 0);

 public bool IsAvailable(IServiceProvider services) => true;

 public IReadOnlyList<IDoctorCheck> GetChecks(DoctorPluginContext context) =>
 [
 new MyCustomCheck(),
 new AnotherCustomCheck()
 ];

 public Task InitializeAsync(DoctorPluginContext context, CancellationToken ct)
 {
 // Optional initialization
 return Task.CompletedTask;
 }
}

7) Telemetry

The engine emits the StellaOps.Doctor.Engine meter with the doctor.check.blocking_probe counter (tags check_id / plugin_id) from CheckExecutor when a check ignores its cancellation token past the effective timeout plus grace window. Discovery timeouts and CanRun failures surface as visible Fail results and Warning logs. Per-service checks are exposed over HTTP only; they emit no Doctor-specific metrics.

Evidence

ClaimSource
Deployable retired; retained sourcedevops/compose/docker-compose.stella-services.yml (RETIRED note), src/Doctor/README.md, src/Doctor/AGENTS.md
Per-service SDK: contract, route, standard and owned-state checks, registrationsrc/__Libraries/StellaOps.Doctor.Plugin.Abstractions/Contract.cs, ServiceDoctorHosting.cs, StandardChecks.cs, OwnedStateChecks.cs, ServiceDoctorRegistrar.cs
Adopting hostsAddServiceDoctorChecks(...) call sites in each service’s Program.cs (for example src/Platform/StellaOps.Platform.WebService/Program.cs, src/JobEngine/StellaOps.JobEngine.WebService/Program.cs)
Registration defaults in the bundledevops/release/bundle/.env.example (*_DOCTOR_REGISTRATION_ENABLED)
Registry and aggregatesrc/Platform/StellaOps.Platform.WebService/Endpoints/DoctorCapabilityEndpoints.cs, Services/DoctorAggregateService.cs
Contractssrc/__Libraries/StellaOps.Doctor.Contracts/
Engine, resilience, telemetry metersrc/__Libraries/StellaOps.Doctor/Engine/CheckExecutor.cs, Engine/CheckRegistry.cs
Analyzersrc/__Analyzers/StellaOps.Doctor.Analyzers/BareCatchAnalyzer.cs
Remediation gatesrc/__Libraries/StellaOps.Doctor.Remediation/RemediationGate.cs, RemediationOptions.cs
Air-gap helperssrc/__Libraries/StellaOps.Doctor/Plugins/DoctorPluginContextExtensions.cs
Estate worker closuresrc/JobEngine/StellaOps.JobEngine.EstateWorker/StellaOps.JobEngine.EstateWorker.csproj
CLIsrc/Cli/StellaOps.Cli/Commands/DoctorCommandGroup.cs, src/Cli/StellaOps.Cli/Program.cs
Plugin librariessrc/__Libraries/StellaOps.Doctor.Plugins.*, src/Doctor/__Plugins/StellaOps.Doctor.Plugin.*