Doctor — Architecture

Audience: operators and SREs diagnosing a Stella Ops deployment, and engineers writing diagnostic plugins.

Stella Doctor is the diagnostic framework that validates system health, configuration, and integration connectivity across the Stella Ops platform. It runs a plugin-based catalog of checks (health, integration, configuration, capability probing, evidence collection) on demand or on a schedule, and is air-gap aware.

The canonical engine, plugin contracts, and result models live in the shared library src/__Libraries/StellaOps.Doctor (namespaces StellaOps.Doctor.Engine, StellaOps.Doctor.Models, StellaOps.Doctor.Plugins). The HTTP surface and report storage live in src/Doctor/StellaOps.Doctor.WebService. Built-in plugins live under src/__Libraries/StellaOps.Doctor.Plugins.* and src/Doctor/__Plugins/StellaOps.Doctor.Plugin.*.

1) Overview

Doctor provides a plugin-based diagnostic system that enables:

• Health checks for all platform components
• Integration validation for external systems (registries, SCM, CI, secrets)
• Configuration verification before deployment
• Capability probing for feature compatibility
• Evidence collection for troubleshooting and compliance

HTTP API surface

StellaOps.Doctor.WebService exposes the following endpoints under the /api/v1/doctor group (see Endpoints/DoctorEndpoints.cs). All require the doctor:run authorization policy except where noted.

Run start is asynchronous: once POST /api/v1/doctor/run returns a run ID, the background run continues independently of the start request lifetime and callers poll GET /api/v1/doctor/run/{runId} for the terminal result.

The service also exposes unauthenticated GET /healthz, GET /readyz, the OpenAPI document, and a build-info endpoint (Program.cs).

Internal compose acceptance endpoints are exposed tenant-free for local operator and CI probes:

These endpoints are a status adapter over the current runtime only. The WebService loads Release, Environment, Scanner, Compliance, BinaryAnalysis, Timestamping, Notify, and EvidenceLocker from mounted bundle manifests under /app/plugins/doctor/<profile>/<plugin-id>. As of commit ee5bd3f9d0 (2026-06-07, WS1 Batch-1), Doctor signature verification and trust-root admission are wired: each mounted bundle is admitted through DoctorMountedRuntimePluginLoader only after a fully-verified detached RSA-PKCS1-SHA256 signature over the assembly bytes (OfflineDevRsaSha256PluginVerifier, AllowUnsigned=false) plus manifest-binding and assembly-SHA-256 checks — so a mounted bundle’s admitted flag now flips true on success and stays false on every reject path. Vex remains not mounted in the base runtime. Timestamping dashboard endpoints remain host-owned, but the checks now reach the Doctor engine through a host facade over the mounted legacy Timestamping check assembly instead of a WebService project reference. See §2 / Mounted runtime plugin loader (signed bundle admission) for the full admission chain, bundle layout, and the docker-compose.plugins.doctor.yml overlay.

Authorization scopes

OAuth scopes are defined in StellaOps.Doctor.WebService.Constants.DoctorScopes and the canonical catalog StellaOps.Auth.Abstractions.StellaOpsScopes:

The standalone Doctor Scheduler host additionally uses doctor-scheduler:read and doctor-scheduler:write (StellaOpsScopes).

Gateway identity-envelope verification. Program.cs registers app.UseIdentityEnvelopeAuthentication() ahead of UseAuthentication() (SPRINT_20260712_009 CLO-7). The JwtBearer OnMessageReceived bridge only promotes an already-hydrated StellaRouterEnvelope principal; on the messaging (Valkey) transport AspNetRouterRequestDispatcher hydrates + HMAC-verifies the envelope before the pipeline runs, but on the HTTP reverse-proxy transport (gateway strips Authorization, forwards only the envelope headers) nothing did, so authenticated calls would 401. The middleware closes that gap and no-ops when the principal is already authenticated (messaging path and direct-Bearer callers are unaffected). Signing key comes from Router:IdentityEnvelopeSigningKey via the shared compose anchor.

Scheduler Integration (Sprint 20260408-003)

The standalone Doctor Scheduler service is deprecated. Doctor health check scheduling is now handled by the Scheduler service’s DoctorJobPlugin.

Doctor schedules are managed via the Scheduler API with jobKind="doctor" and plugin-specific configuration in pluginConfig. Trend data is stored in scheduler.doctor_trends (PostgreSQL).

Scheduler-hosted Doctor endpoints:

• GET /api/v1/scheduler/doctor/trends – aggregated trend summaries
• GET /api/v1/scheduler/doctor/trends/checks/{checkId} – per-check trend data
• GET /api/v1/scheduler/doctor/trends/categories/{category} – per-category trend data
• GET /api/v1/scheduler/doctor/trends/degrading – checks with degrading health

Schedule management uses the standard Scheduler API at /api/v1/scheduler/schedules with jobKind="doctor" and pluginConfig containing Doctor-specific options (mode, categories, alerts). When pluginConfig.alerts.enabled is true, the live Scheduler Doctor job keeps the run-log evidence line and, for matching alertOnFail / alertOnWarn outcomes, publishes a tenant-scoped Notify queue event with kind scheduler.doctor.run_alert. The payload includes the schedule ID, Doctor run ID, aggregate pass/warn/fail counts, health score, configured alert channels, and the failed/warn check IDs. If the local Notify event queue is not wired, the Scheduler job does not throw or call an external sink; it records an explicit log-only fallback in the run log.

Three default Doctor schedules are seeded by SystemScheduleBootstrap:

• doctor-full-daily (0 4 * * *) – Full health check
• doctor-quick-hourly (0 * * * *) – Quick health check
• doctor-compliance-weekly (0 5 * * 0) – Compliance category audit

The Doctor WebService (src/Doctor/StellaOps.Doctor.WebService/) remains the execution engine. The plugin communicates with it via HTTP POST to /api/v1/doctor/run.

The legacy src/Doctor/StellaOps.Doctor.Scheduler/ host is retained only for Development/Testing. It now fails fast outside Development/Testing so live runtime cannot silently fall back to the deprecated volatile schedule/trend repositories.

Report Storage Runtime

Doctor WebService persists historical reports through PostgresReportStorageService when ConnectionStrings:StellaOps or Database:ConnectionString is configured. The report store owns the doctor PostgreSQL schema and the doctor.doctor_reports table (created by Migrations/001_report_storage.sql: run_id PK, started_at, completed_at, overall_severity, the per-severity count columns passed_count/warning_count/failed_count/skipped_count/info_count/total_count, a gzip-compressed report_json_compressed BYTEA blob, and created_at). StellaOps.Doctor.WebService embeds Migrations/**/*.sql and registers startup migrations for the Doctor.ReportStorage module (AddStartupMigrations<…> in DoctorReportStorageExtensions), so fresh databases converge before report endpoints can write. Missing report-storage database configuration fails startup in every runtime environment (AddDoctorReportStorage throws when neither connection string is set). Reports are retained for Doctor:ReportRetentionDays days (default 30); a background timer prunes older rows hourly. The volatile report store is a test-project fixture only and is not registered by Doctor WebService production code.

AdvisoryAI Diagnosis Surface

Doctor WebService exposes a diagnosis endpoint for AdvisoryAI-backed health analysis:

• POST /api/v1/doctor/diagnosis

The endpoint accepts either:

• runId referencing a stored Doctor report, or
• an inline DoctorRunResultResponse payload.

It maps the run through the shared Doctor AdvisoryAI context adapter (src/__Libraries/StellaOps.Doctor/AdvisoryAI/**) and returns deterministic diagnosis output (issues, root causes, recommended actions, and related runbook links).

Runtime wiring includes:

• IDoctorContextAdapter for deterministic context projection from Doctor reports
• IDoctorAIDiagnosisService (deterministic implementation) for assessment, root cause, correlation, and remediation projection
• a built-in IEvidenceSchemaRegistry seeded with the common diagnosis schemas during service registration; live runtime no longer binds the mutable in-memory registry type

Air-Gap Awareness (Sprint 20260501-030)

Doctor checks that perform external network probes are air-gap aware. When the deployment is sealed (or the operator has forced offline mode pre-seal), checks short-circuit with a canonical message rather than emitting Fail against unreachable endpoints.

Signals consulted (in order):

1. Doctor:OfflineProfile config key (env override STELLAOPS_DOCTOR_OFFLINE_PROFILE=true) — read once in Program.cs and folded into the configuration root. Useful for disconnected commissioning before AirGap sealing.
2. IEgressPolicy.IsSealed from StellaOps.AirGap.Policy — the platform’s authoritative sealed-mode signal.

Helpers:

• DoctorPluginContextExtensions.IsAirgapped(this DoctorPluginContext) — single entry point for context-driven plugins (Attestor, Auth, BinaryAnalysis, Notify, Observability).
• DoctorPluginContextExtensions.IsLoopbackOrPrivateAddress(string?) — used by Notify/Observability checks so internal collectors (loopback, RFC1918, link-local, IPv6 ULA/link-local) are still probed in sealed mode. No DNS resolution is performed (that would itself be an external probe).
• StellaOps.Doctor.Plugin.Timestamping.AirGapAwareness — companion helper for the Timestamping plugin, which uses constructor-injected IEgressPolicy? rather than DoctorPluginContext-resolved DI.

Canonical Skip wording: "air-gap mode: external probe disabled" (DoctorPluginContextExtensions.SealedSkipReason). Each gated check additionally emits an evidence row airGapSealed=true so operators can self-explain Skip results.

Gated checks (the enumerated list below is authoritative; an absolute count rots as checks are added — do not cite one):

• Plugin.Attestor: RekorConnectivityCheck, RekorClockSkewCheck, TransparencyLogConsistencyCheck
• Plugin.Auth: OidcProviderConnectivityCheck
• Plugin.BinaryAnalysis: DebuginfodAvailabilityCheck
• Plugin.Notify: SlackConnectivityCheck, TeamsConnectivityCheck, WebhookConnectivityCheck, EmailConnectivityCheck, NotifyChannelConnectivityCheck (with internal-endpoint allowance for self-hosted Webhook/SMTP and aggregate setup-wizard compatibility)
• Plugin.Observability: OtlpEndpointCheck, PrometheusScrapeCheck (allow loopback/RFC1918 collectors in sealed mode)
• Plugin.Timestamping: TimeSkewChecks (NTP), TsaAvailabilityCheck, TsaCertificateExpiryCheck, EuTrustListChecks, CrlDistributionCheck, OcspResponderCheck, RevocationCacheFreshCheck

Cache TTL defaults (Timestamping/PKI offline-friendly artefacts, declared in TimestampingCacheOptions):

Startup banner: at startup, StellaOps.Doctor.Startup logger emits effective, source (forced | from airgap | disabled), airgap.IsSealed, and the env-var value once so operators can verify the resolved offline profile.

Offline-kit artefact registry (delivered 2026-05-02): StellaOps.AirGap.Importer.Artefacts.IAirGapArtefactStore exposes typed lookups for the offline-kit-shipped artefacts the Timestamping plugin needs (TSA chains keyed by qTSP identifier, EU LOTL + member-state TSL XML, CRL snapshots keyed by SHA-256 of the distribution-point URI, OCSP responses keyed by responder URI + cert-id hash, and an NTP stratum proof). The default FilesystemAirGapArtefactStore is rooted at var/lib/stellaops/airgap/timestamping/ and reads the registry-level manifest.json once at construction; every lookup re-verifies SHA-256 against the manifest entry, so manifest tampering surfaces as a missing artefact (fail-closed for the consumer). The six gated Timestamping checks now consult this registry under sealed mode and emit canonical evidence rows on success: airGapSealed=true, source=offline-kit, artefactSha256=…, artefactIssuedAt=…. When the configured artefact is absent, the check emits Unhealthy + source=offline-kit-missing with an explicit remediation path — sealed mode without the offline kit must NOT silently pass.

2) Plugin Architecture

The interfaces below are the canonical contracts in StellaOps.Doctor.Plugins (Plugins/IDoctorPlugin.cs, Plugins/IDoctorCheck.cs, Plugins/DoctorPluginContext.cs) and StellaOps.Doctor.Models. A separate, simpler IDoctorPlugin (Name/Categories/RunChecksAsync) exists in StellaOps.Doctor.Plugins.Core for legacy aggregation; the engine and all built-in plugins use the contracts shown here.

Core Interfaces

public interface IDoctorPlugin
{
    string PluginId { get; }          // e.g. "stellaops.doctor.integration"
    string DisplayName { get; }
    DoctorCategory Category { get; }  // enum, not string
    Version Version { get; }
    Version MinEngineVersion { get; }

    bool IsAvailable(IServiceProvider services);
    IReadOnlyList<IDoctorCheck> GetChecks(DoctorPluginContext context);
    Task InitializeAsync(DoctorPluginContext context, CancellationToken ct);
}

public interface IDoctorCheck
{
    string CheckId { get; }           // e.g. "check.integration.oci.referrers"
    string Name { get; }
    string Description { get; }
    DoctorSeverity DefaultSeverity { get; }
    IReadOnlyList<string> Tags { get; }
    TimeSpan EstimatedDuration { get; }

    bool CanRun(DoctorPluginContext context);
    Task<DoctorCheckResult> RunAsync(DoctorPluginContext context, CancellationToken ct);
}

DoctorCategory is an enum (Core, Database, Security, Cryptography, Attestation, Docker, Integration, Observability, Notify, ServiceGraph, Sources, Verification, Authority, AI, Environment, Release, Scanner, Infrastructure).

Plugin Context

public sealed class DoctorPluginContext
{
    public required IServiceProvider Services { get; init; }
    public required IConfiguration Configuration { get; init; }
    public required TimeProvider TimeProvider { get; init; }
    public required ILogger Logger { get; init; }
    public required string EnvironmentName { get; init; }
    public string? TenantId { get; init; }
    public IConfiguration PluginConfig { get; init; }   // bound to Doctor:Plugins

    // Factory for building results, and a static credential redactor.
    public DoctorCheckResultBuilder CreateResult(string checkId, string pluginId, string category);
    public static string Redact(string? value);
}

Check Results

The result type returned by a check is DoctorCheckResult (StellaOps.Doctor.Models). The simpler CheckResult record (CheckId/Severity/IsHealthy/Message/Metadata) is a separate dashboard projection produced by DoctorEngine.RunChecksAsync.

public sealed record DoctorCheckResult
{
    public required string CheckId { get; init; }
    public required string PluginId { get; init; }
    public required string Category { get; init; }
    public required DoctorSeverity Severity { get; init; }
    public required string Diagnosis { get; init; }
    public required Evidence Evidence { get; init; }
    public IReadOnlyList<string>? LikelyCauses { get; init; }
    public Remediation? Remediation { get; init; }
    public string? VerificationCommand { get; init; }
    public required TimeSpan Duration { get; init; }
    public required DateTimeOffset ExecutedAt { get; init; }
}

public enum DoctorSeverity
{
    Pass = 0,   // Check passed successfully
    Info = 1,   // Informational (not a problem)
    Warn = 2,   // Warning condition that should be addressed
    Fail = 3,   // Failure (needs attention)
    Skip = 4    // Skipped (not applicable in current context)
}

Evidence carries a Description, an IReadOnlyDictionary<string, string> Data, and an optional SensitiveKeys list. Remediation carries ordered Steps (each with a CommandType of Shell/Sql/Api/FileEdit/Manual), a SafetyNote, RequiresBackup, and an optional RunbookUrl.

Engine resilience (ADR-026 D8)

The executor (Engine/CheckExecutor.cs) and registry (Engine/CheckRegistry.cs) are hardened so a single misbehaving check or plugin can never crash, stall, or silently shrink a run. See ADR-026 decisions D8 (engine resilience) and D6 (rejected/lost checks must not read as all-green).

• CanRun isolation. check.CanRun(context) is invoked inside the per-check guarded region. A throwing CanRun (e.g. it dereferences an unregistered service or reads a missing config value) becomes a structured Fail result whose diagnosis/evidence names CanRun as the failing stage (Evidence.Data["Stage"] = "CanRun"), instead of escaping ExecuteCheckAsync and aborting the whole Parallel.ForEachAsync body. A normal false return still yields the existing Skip “not applicable” result.

• Per-check timeout sizing. The effective per-check timeout is derived from IDoctorCheck.EstimatedDuration as clamp(max(globalTimeout, EstimatedDuration × 3), globalTimeout, 5 min). A check with an unset/zero estimate gets exactly the global DoctorRunOptions.Timeout (unchanged behaviour); a legitimately slow check (large estimate) is granted proportional headroom so it is not falsely timed out, bounded by a 5-minute ceiling so a garbage estimate cannot pin a slot indefinitely.

• Cooperative-timeout / blocking-probe watchdog. CancelAfter only signals the token; a check that ignores its CancellationToken and blocks (busy loop, blocking sync I/O) would otherwise pin one of the Parallelism slots for the full hang. The executor dispatches RunAsync to the thread pool and races it against Task.Delay(effectiveTimeout + grace). A check that honours its token unwinds within the grace window and produces the existing clean-timeout Fail. A check that ignores its token loses the race and is reported as a distinct blocking-probe Fail (diagnosis “blocking probe (ignored cancellation)”, Evidence.Data["BlockingProbe"] = "true") without awaiting the stuck task — it is left to run orphaned, and a structured warning plus the doctor.check.blocking_probe metric (meter StellaOps.Doctor.Engine, tagged by check_id/plugin_id) is emitted. Known limitation: modern .NET cannot safely abort a thread that ignores its token, so the contract is detect-and-report — the orphaned probe still occupies its parallel slot until it returns; the run as a whole completes instead of hanging.

• Discovery-time timeout. Per-plugin IsAvailable and GetChecks discovery calls each run under a bounded DoctorRunOptions.DiscoveryTimeout (default 10s, distinct from the per-check Timeout). The synchronous call is dispatched to the thread pool and awaited up to the discovery timeout; a plugin that blocks in either is skipped for that run rather than stalling discovery before any check executes, and a hang in one plugin does not delay the others. A timed-out plugin is not silently dropped: the registry injects a synthetic failing check (check.core.plugins.discovery-timeout.<pluginId>, severity Fail) so the run goes red — closing the D6 “rejected plugin → checks vanish → all green” hole. The existing per-plugin try/catch for throwing plugins is preserved.

• Opt-in probe retry (ProbeRetry). A check that performs a single network/IPC attempt and fails it on a transient blip (a dropped connection, a momentary DNS hiccup) would otherwise report a false Fail. The shared StellaOps.Doctor.Resilience.ProbeRetry.ExecuteAsync<T>(probe, options, timeProvider, jitter, logger, ct) helper wraps an outbound probe in a bounded retry-with-full-jitter, distinct from and composed within the per-check timeout (the timeout bounds the whole check; retry repeats the probe call inside it). It is opt-in — a non-idempotent / side-effecting probe must not opt in — and is a standalone primitive that does not depend on IDoctorCheck or the IntegrationCheck base; the D2 IntegrationCheck base (sprint SPRINT_20260608_005_*) advertises a retry-policy descriptor and wraps its probe body in this helper. Semantics:

  • Retries only thrown transient failures, classified via CheckExceptionLogging.IsTransientProbeException (HTTP / socket / IO / probe-timeout). A non-transient exception (JsonException/contract/programming error) is rethrown on the first attempt and never retried.
  • A successful return is returned immediately — even a “Fail-shaped” value. The helper retries thrown transient exceptions of the probe call, not result severities.
  • Bounded attempts (ProbeRetryOptions.MaxAttempts, default 3) with exponential backoff (BaseDelay, default 200 ms; doubling, capped by MaxDelay, default 5 s) and full jitter — the delay is sampled uniformly in [0, ceiling]. Jitter randomness comes from an injected IJitterSource (CryptoJitterSource in production, a fixed/fake source in tests — never Random.Shared, per the Doctor coding standard), and backoff delays use the injected TimeProvider (Task.Delay(delay, timeProvider, ct)) so tests run on virtual time and stay fast/offline.
  • Cancellation from the supplied token propagates as OperationCanceledException and is never treated as a transient retry (even though TaskCanceledException is otherwise in the transient set); retrying stops immediately on cancel.
  • On attempt exhaustion the last transient exception is rethrown (stack preserved) so the caller’s check turns it into a Fail via the existing engine/timeout path. The returned ProbeRetryResult<T> carries the attempt count so the check can record it in result evidence; the total retry budget is bounded by the caller’s ct (the engine ties it to the DOC-ENG-003 effective per-check timeout, so it never exceeds the per-check ceiling).
  • Current DOC-OBS-005 audit. The IntegrationCheck base supplies the default bounded retry policy to check.db.connection, check.postgres.schemas.present, check.servicegraph.valkey, check.router.messaging.transport, check.servicegraph.nats, check.servicegraph.gateway-router, check.servicegraph.sibling-reachability, check.integration.oci.registry, check.integration.oci.credentials, check.integration.oci.pull, check.integration.oci.referrers, check.integration.git, check.integration.oidc, check.security.authority-signing-key, and check.smremote.remote-hsm.connectivity. The Observability OTLP and Prometheus reachability checks override a shorter two-attempt policy for local collector/scrape probes. The log-shipping check intentionally stays single-attempt because it reads one local forwarder-state snapshot and reports the returned drain/backlog state; retrying a returned stalled/unavailable snapshot would hide operator-visible state. check.integration.oci.push is also intentionally single-attempt because it initiates a registry upload session and cleanup; retry/fan-out would need a separate idempotency design before it can safely repeat that write-shaped probe.

• Tenant fan-out for tenant-scoped IntegrationChecks. Tenant-scoped integration checks enumerate IDoctorTenantCatalog before probing. The library default catalog is read-only and local-configuration backed (Doctor:TenantCatalog:Tenants, with Doctor:RequiredTenants / Authority:RequiredTenants fallbacks) for offline tests and embedded hosts. The Doctor WebService replaces that seam with a read-only shared.tenants catalog when a platform database connection is configured (Doctor:TenantCatalog:ConnectionString, ConnectionStrings:Platform, Platform:Storage:PostgresConnectionString, ConnectionStrings:StellaOps, or Database:ConnectionString). The production catalog reads active rows from shared.tenants; it does not create, update, reconcile, or backfill tenants. If no catalog is registered, the catalog is empty, or exactly one tenant is cataloged, the check preserves historical single-run behavior and evidence shape. When two or more tenants are present, the base runs the probe once per cataloged tenant with DoctorPluginContext.TenantId set, resolves tenant overrides before global config (Doctor:TenantOverrides:<tenant>:<key> then Tenants:<tenant>:<key>), and emits aggregate evidence: TenantCount, TenantResults, TenantProbeAttempts, TenantProbeFailures, and per-tenant severity/retry-attempt fields. Aggregate severity is worst-result: any tenant fail fails the aggregate, so a green default tenant no longer masks a failing second tenant.

  • Current DOC-OBS-001 audit. Tenant fan-out is enabled for the tenant-scoped integration checks that already derive from IntegrationCheck: check.db.connection, check.postgres.schemas.present, check.servicegraph.valkey, check.router.messaging.transport, check.integration.oci.registry, check.integration.oci.credentials, check.integration.oci.pull, check.integration.oci.referrers, check.integration.git, check.integration.oidc, and check.platform.identity-envelope. Platform/shared-state checks such as gateway/router, NATS, sibling reachability, circuit-breaker state, connection-pool state, Authority JWKS, and telemetry backends remain single-probe. check.integration.oci.push also remains outside this fan-out because its probe opens a registry upload session.

Bare-catch analyzer scope (DOCTOR0001, ADR-026 D2/D7)

The BareCatchAnalyzer (src/__Analyzers/StellaOps.Doctor.Analyzers/BareCatchAnalyzer.cs) flags unfiltered catch { } / catch (Exception) so probe failures cannot be silently swallowed (the canonical fix is the catch (Exception ex) when (CheckExceptionLogging.Is…Exception(ex)) pattern; see §4 Check Patterns and src/Doctor/AGENTS.md).

• Scope (DOC-ENG-005): the analyzer fires inside types implementing StellaOps.Doctor.Plugins.IDoctorCheck and inside the non-check delegate-helper classes a check directly invokes (a method call, member access, or construction of a source-defined non-check type from within a check). A swallow in a probe-helper the analyzer previously could not see is now caught. Detection is sound — only helpers provably reached from a check fire, so unrelated handlers/utilities in the same assembly stay silent — and compilations that do not reference the IDoctorCheck symbol early-exit at zero added cost, keeping non-Doctor csprojs unaffected.
• Severity: Error. DOC-DEBT-001 completed the Phase-2 flip on 2026-06-11: DOCTOR0001 now ships as DiagnosticSeverity.Error and is absent from the WarningsNotAsErrors shim in src/Directory.Build.props. The final blocker was the remaining in-process AI plugin raw catches (LocalInferenceCheck and OllamaProviderCheck), now narrowed to typed parse catches with CheckExceptionLogging. A full Doctor-surface build (Plugins.*, mounted Plugin.*, engine, WebService, Scheduler) passed with DOCTOR0001 at Error before the shim was removed. No project-level suppression should be added to bypass this gate.
• Deferred — swallow-to-Pass: a rule to flag returning a Pass-shaped DoctorCheckResult from inside a catch (the ADR-026 D7 fake-green) was scoped but deferred — distinguishing a fabricated Pass from a legitimately-justified one is a semantic judgment a syntactic analyzer cannot make without false positives. It stays enforced by code review and the no-hardcoded-Pass test discipline.
• Documented Timestamping exception: StellaOps.Doctor.Plugin.Timestamping still declares its own local IDoctorCheck for the mounted legacy facade, so the canonical-interface gate does not directly cover those check types even though the project references shared Doctor helpers. The current Timestamping raw-catch sites have been manually narrowed to typed handlers and verified by project build; the durable fix remains migrating the plugin onto StellaOps.Doctor.Plugins.IDoctorCheck.

3) Built-in Plugins

StellaOps.Doctor.WebService registers the following plugins at startup (Program.cs). Each plugin exposes one or more IDoctorCheck implementations.

Additional Doctor plugin projects live under src/Doctor/__Plugins/. The directory contains Plugin.Agent, Plugin.BinaryAnalysis, Plugin.Compliance, Plugin.Environment, Plugin.EvidenceLocker, Plugin.Notify, Plugin.Operations, Plugin.Release, Plugin.Scanner, Plugin.Storage, Plugin.Timestamping, plus Plugins.Agent and Plugins.Core. Most of these are the sources of the mounted-bundle checks already listed above (Release, Environment, Scanner, Compliance, BinaryAnalysis, Timestamping, Notify, EvidenceLocker, Operations). The remainder (Plugin.Agent / Plugins.Agent, Plugin.Storage) are not registered by the WebService Program.cs; they are consumed by other hosts/tests — treat them as available components rather than part of the default WebService catalog. (There is no Auth, Crypto, Policy, Postgres, or Vex directory here; Cryptography and Authority are the in-process __Libraries plugins listed in the table above.) The mounted Plugin.Operations checks do not own queue or scheduler recovery: they read a JobEngine-owned IOperationsHealthProbe when a host registers one and otherwise return an explicit Skip(reason) instead of fixture Pass data. Their heal posture is ManualOnly until JobEngine wires safe queue/scheduler remediation through the gated Doctor remediation engine.

ServiceGraph sibling-service reachability

StellaOps.Doctor.Plugins.ServiceGraph includes check.servicegraph.sibling-reachability for the first-party sibling-service catalog family (integration.service.*). The legacy check.servicegraph.endpoints still probes direct StellaOps:*Url values when standalone deployments configure them, but the platform Doctor deployment normally configures Router:Gateways:0:Host instead. The sibling reachability check therefore binds to integration.gateway.router, composes the gateway base URL, and sends read-only GET probes through the documented router route roots for Authority, Concelier, Scanner, Policy, Evidence Locker, Attestor, JobEngine, Notify, Release Orchestrator, VexLens, and VexHub.

Outcomes are deterministic: no router host configured is Skip; every route returning HTTP 2xx is Pass; any non-2xx response or transient probe exception is Fail with UnhealthyServices plus per-service URL/status evidence. The integration catalog marks each integration.service.* entry as owned by this check so check.core.integration-coverage no longer reports sibling services as uncovered solely because direct per-service URLs are absent.

Final integration-catalog owner bindings

The declared integration catalog now has owning checks for the former non-sibling null-owner tail:

• integration.transparency.fulcio is owned by check.attestation.cosign.keymaterial. In keyless Sigstore mode that check probes Fulcio with GET /api/v2/configuration and fails visibly on an unhealthy status; non-keyless modes report their own signing posture instead of claiming Fulcio reachability.
• integration.mirror.debuginfod is owned by check.binaryanalysis.debuginfod.available. That check probes configured DEBUGINFOD_URLS endpoints, falls back to the default public debuginfod endpoint only outside sealed mode, and returns the canonical air-gap Skip(reason) when external probing is disabled.

The source-level coverage gate (check.core.integration-coverage) is green when every declared owning check id is present in the live check-id set. A release/closeout proof still needs a live Doctor run against the local stack so mounted-plugin admission and runtime registration are proven from the running service, not just from the declared manifest and unit-level check catalog.

Runtime plugin composition gap

The WebService no longer compiles optional Doctor implementations directly. Program.cs calls AddMountedDoctorRuntimePlugins(builder.Configuration), which reads Doctor:RuntimePlugins:RootPath (default /app/plugins/doctor) and Doctor:RuntimePlugins:Profile (default base), then registers manifest/checksum-valid runtime targets from mounted bundle directories.

The current scoped project-reference audit for StellaOps.Doctor.WebService.csproj has no src/Doctor/__Plugins implementation violations. The only remaining Doctor audit entry is the deferred core Router messaging bridge, which is not a Doctor plugin implementation.

The 2026-06-06 Doctor mounted-loader passes removed StellaOps.Doctor.Plugin.Release, StellaOps.Doctor.Plugin.Environment, StellaOps.Doctor.Plugin.Scanner, StellaOps.Doctor.Plugin.Compliance, StellaOps.Doctor.Plugin.BinaryAnalysis, and StellaOps.Doctor.Plugin.Timestamping from the WebService project graph. The loader consumes the existing runtime-bundle manifest shape (manifest.json with id, module=doctor, profile, contractVersion=runtime-bundle.v1, capabilities=["doctor:checks"], and assembly.path/sha256).

The 2026-06-07 follow-up (commit ee5bd3f9d0, WS1 Batch-1) closed the remaining gap: DoctorMountedRuntimePluginLoader now verifies the detached <assembly>.sig against the mounted Doctor trust root before any bundle code is activated, so a mounted plugin’s admitted flag flips true only on a fully-verified bundle and stays false on every reject path. Current behavior is mounted-first: Program.cs registers runtime plugins from mounted manifests when present and reports them as not-mounted (root/manifest absent) or rejected (any admission-chain failure) when absent or invalid. Vex remains a target mounted Doctor bundle, but the WebService does not compile StellaOps.Doctor.Plugin.Vex because it is not registered by Program.cs; /internal/plugins/status and /internal/plugins/probe continue to report it as not mounted until a signed mounted Vex bundle is admitted.

Live runtime probe is pending. The admission code, the 12 loader tests (including a live forcing-function: signature tamper → reject, restore → admit), and the docker-compose.plugins.doctor.yml overlay are committed, but a live overlay-up acceptance probe against a running doctor-web (mount the signed bundle, confirm /internal/plugins/probe reports the admitted Release check) has not yet been recorded.

Mounted runtime plugin loader (signed bundle admission)

Doctor’s mounted-bundle loader (src/Doctor/StellaOps.Doctor.WebService/Services/DoctorMountedRuntimePluginLoader.cs, commit ee5bd3f9d0) is wired into Program.cs via AddMountedDoctorRuntimePlugins(builder.Configuration). It is fail-closed: a bundle is only registered after its assembly clears the full admission chain, and every reject path registers zero plugins (asserted by tests).

Admission chain (per bundle directory, in order):

1. Manifest binding — manifest.json must exist and parse; id must equal the bundle directory name; module must be doctor; contractVersion must be runtime-bundle.v1; the manifest must declare capability doctor:checks; the configured profile (when non-empty) must match; and a well-formed assembly descriptor (relative path + sha256) must be present.
2. Path-traversal guard + assembly SHA-256 — the assembly path must be bundle-relative (not rooted, no .. escape) and the on-disk assembly bytes must hash to the manifest sha256.
3. Detached signature verification — DoctorMountedRuntimePluginLoader verifies <assembly>.sig with OfflineDevRsaSha256PluginVerifier (detached RSA-PKCS1-SHA256 over the assembly bytes) bound to the configured trust root, with AllowUnsigned=false. Only then is the bundle’s IDoctorPlugin (or the Timestamping legacy-check facade) activated.

Configuration (Doctor:RuntimePlugins, env prefix DOCTOR_):

Release-plugin checks additionally require ReleaseOrchestrator:Url. The canonical Compose stack sets it to the backend-network service address (http://release-orchestrator.stella-ops.local:8080); without that explicit value the Release checks correctly remain unavailable instead of probing an implicit or external endpoint.

The canonical Compose stack also sets Doctor:Evidence:Root to /var/lib/stellaops/doctor-evidence and mounts the doctor-evidence named volume there. Doctor runs therefore retain their deterministic JSONL evidence while the non-root service keeps /app read-only; using the content-root fallback (/app/artifacts) in the container is not writable. The one-shot doctor-evidence-init service converges the volume to uid/gid 10001:10001 before doctor-web starts, including after a fresh volume is created.

Bundle / trust-root layout:

Bundle producer: devops/build/package-runtime-plugins.ps1 -Module doctor -Profile base -SignDoctorBundles -UseOfflineDevSigner stages the signed Doctor bundles (including stellaops.doctor.release) and produces the offline-dev trust root + cosign.pub. Both switches are required: -SignDoctorBundles enables signing, while -UseOfflineDevSigner selects the local RSA/SHA-256 signer. Omitting the former emits unsigned bundles that the fail-closed loader rejects. The generated cosign.pub is git-ignored under the doctor trust-root directory.

Opt-in compose overlay: devops/compose/docker-compose.plugins.doctor.yml layers read-only mounts of the signed bundle + trust root onto doctor-web and restates the loader defaults (Doctor__RuntimePlugins__RootPath/Profile/TrustRootPath, AllowUnsigned=false) so the mount↔config contract is explicit and immune to default drift. Apply with COMPOSE_EXTRA_FILES=docker-compose.plugins.doctor.yml ./scripts/compose-cli.ps1 up.

Probe / status surface: mounted bundles surface in the canonical PluginProbeReport at GET /internal/plugins/status (no probe execution) and POST /internal/plugins/probe (deterministic dry-run). Admitted bundles report admitted=true; rejected bundles surface with their rejection reason — there is no silent-green path.

Target behavior remains signed mounted bundle loading for every extension target: read-only Doctor bundles are staged under /app/plugins/doctor/<profile>/<plugin-id>, with profile/config/trust material separated from the base image. The /internal/plugins/status and /internal/plugins/probe endpoints remain red for extension targets that have no signed mounted bundle yet (Vex, and any not-yet-staged target) until mounted equivalents are signature-admitted, loaded, and probed from the bundle root. Custom third-party Doctor plugins become compose-ready once a signed bundle is staged into the mount root and the operator mounts the trust root.

IntegrationPlugin (stellaops.doctor.integration)

Validates external system connectivity and capabilities (StellaOps.Doctor.Plugins.Integration.IntegrationPlugin). Selected checks:

Each check’s runtime severity is determined when it executes (via the result builder), not by a fixed catalog severity. Use GET /api/v1/doctor/checks to enumerate the live catalog including DefaultSeverity, Tags, and EstimatedDurationMs.

See Registry Diagnostic Checks for detailed documentation of the OCI registry checks.

4) Check Patterns

Non-Destructive Probing

Registry checks use non-destructive operations:

// Pull check: HEAD request only (no data transfer)
var response = await client.SendAsync(new HttpRequestMessage(HttpMethod.Head, manifestUrl), ct);

// Push check: Start upload then immediately cancel
var uploadResponse = await client.PostAsync(uploadsUrl, null, ct);
if (uploadResponse.StatusCode == HttpStatusCode.Accepted)
{
    var location = uploadResponse.Headers.Location;
    await client.DeleteAsync(location, ct); // Cancel upload
}

Capability Detection

Registry capability probing sequence:

1. GET /v2/ → Extract OCI-Distribution-API-Version header
2. GET /v2/{repo}/referrers/{digest} → Check referrers API support
3. POST /v2/{repo}/blobs/uploads/ → Check chunked upload support
   └─ DELETE {location} → Cancel upload session
4. POST /v2/{repo}/blobs/uploads/?mount=...&from=... → Check cross-repo mount
5. OPTIONS /v2/{repo}/manifests/{ref} → Check delete support (Allow header)
6. OPTIONS /v2/{repo}/blobs/{digest} → Check blob delete support

Evidence Collection

All checks collect structured evidence via the fluent DoctorCheckResultBuilder created from the context. WithEvidence(description, configure) takes the evidence description plus a callback over EvidenceBuilder:

var result = context.CreateResult(CheckId, PluginId, Category)
    .Pass("Registry authentication successful")
    .WithEvidence("OCI registry authentication probe", eb => eb
        .Add("registry_url", registryUrl)
        .Add("auth_method", "bearer")
        .Add("response_time_ms", (int)elapsed.TotalMilliseconds)
        .AddSensitive("token_preview", DoctorPluginContext.Redact(token)))
    .Build();

EvidenceBuilder.Add has overloads for string, bool, int, long, and double. AddConnectionString redacts the value automatically and marks the key sensitive; AddSensitive marks an already-redacted value sensitive.

Credential Redaction

Sensitive values are redacted by the static DoctorPluginContext.Redact helper (Plugins/DoctorPluginContext.cs). It shows the first 3 and last 3 characters for values longer than 8 characters; shorter or empty values collapse to a fixed token:

public static string Redact(string? value)
{
    if (string.IsNullOrWhiteSpace(value))
        return "[EMPTY]";
    if (value.Length <= 8)
        return "[REDACTED]";
    return $"{value[..3]}***{value[^3..]}";
}
// "mysecretpassword" → "mys***ord"
// "short"            → "[REDACTED]"

5) CLI Integration

The stella doctor command group (StellaOps.Cli.Commands.DoctorCommandGroup) runs the engine in-process. Subcommands: run (also the default action), list, export, fix, and suggest.

# Run all checks (default action == `doctor run`)
stella doctor

# Run mode: quick | normal (default) | full
stella doctor --mode quick
stella doctor --mode full

# Run checks filtered by category or tag
stella doctor --category Integration
stella doctor --tag quick --tag connectivity   # repeatable

# Run a specific check by ID
stella doctor --check check.integration.oci.referrers

# Output formats: text (default), json, markdown (alias: md)
stella doctor --format json
stella doctor --format markdown

# Write output to a file
stella doctor --output report.json --format json

# Execution controls
stella doctor --parallel 8        # max parallel checks (default 4)
stella doctor --timeout 60        # per-check timeout in seconds (default 30)
stella doctor --fail-on-warn      # non-zero exit on warnings (default: fail only on errors)
stella doctor --watch --interval 60   # continuous monitoring mode
stella doctor --env prod          # target environment hint
stella doctor --verbose           # per-check progress + extra detail

# List available checks
stella doctor list --category Integration --tag quick

# Generate a support bundle (ZIP)
stella doctor export --output bundle.zip --include-logs --log-duration 4h

# Apply non-destructive fixes from a JSON report (dry-run unless --apply)
stella doctor fix --from report.json --apply

Exit codes are set from the run’s overall severity: failures map to CliExitCodes.DoctorFailed; warnings map to CliExitCodes.DoctorWarning only when --fail-on-warn is set. doctor fix only auto-executes safe stella shell commands without placeholders and without requiresBackup.

6) Extensibility

Creating a Custom Check

public sealed class MyCustomCheck : IDoctorCheck
{
    public string CheckId => "check.custom.mycheck";
    public string Name => "My Custom Check";
    public string Description => "Validates custom integration";
    public DoctorSeverity DefaultSeverity => DoctorSeverity.Fail;
    public IReadOnlyList<string> Tags => ["custom", "integration"];
    public TimeSpan EstimatedDuration => TimeSpan.FromSeconds(5);

    public bool CanRun(DoctorPluginContext context)
    {
        // Return false if preconditions not met
        return context.Configuration["Custom:Enabled"] == "true";
    }

    public async Task<DoctorCheckResult> RunAsync(DoctorPluginContext context, CancellationToken ct)
    {
        var builder = context.CreateResult(CheckId, "custom", "Integration");

        try
        {
            // Perform check logic
            var result = await ValidateAsync(context, ct);

            if (result.Success)
            {
                return builder
                    .Pass("Custom validation successful")
                    .WithEvidence("Custom validation", eb => eb.Add("detail", result.Detail))
                    .Build();
            }

            return builder
                .Fail("Custom validation failed")
                .WithCauses("Configuration is invalid")
                .WithRemediation(rb => rb
                    .AddManualStep(1, "Check configuration", "Verify Custom:Setting is correct")
                    .WithRunbookUrl("https://docs.stella-ops.org/runbooks/custom-check"))
                .Build();
        }
        catch (Exception ex)
        {
            return builder
                .Fail($"Check failed with error: {ex.Message}")
                .WithEvidence("Check exception", eb => eb.Add("exception_type", ex.GetType().Name))
                .Build();
        }
    }
}

Creating a Custom Plugin

public sealed class MyCustomPlugin : IDoctorPlugin
{
    public string PluginId => "custom";
    public string DisplayName => "Custom Checks";
    public DoctorCategory Category => DoctorCategory.Integration;  // enum
    public Version Version => new(1, 0, 0);
    public Version MinEngineVersion => new(1, 0, 0);

    public bool IsAvailable(IServiceProvider services) => true;

    public IReadOnlyList<IDoctorCheck> GetChecks(DoctorPluginContext context) =>
    [
        new MyCustomCheck(),
        new AnotherCustomCheck()
    ];

    public Task InitializeAsync(DoctorPluginContext context, CancellationToken ct)
    {
        // Optional initialization
        return Task.CompletedTask;
    }
}

7) Telemetry

StellaOps.Doctor.WebService wires OpenTelemetry via AddStellaOpsTelemetry (Program.cs) with service name StellaOps.Doctor and registers a single meter, StellaOps.Doctor.Runs, for metrics export. Standard ASP.NET Core and HTTP-client instrumentation is included by the shared telemetry package.

NOT IMPLEMENTED: the previously documented custom instruments (doctor_check_duration_seconds, doctor_check_results_total, doctor_plugin_load_duration_seconds) and spans (doctor.run, doctor.check.{check_id}) are not present in the Doctor source. The StellaOps.Doctor.Runs meter is registered but no named counters/histograms were found emitting to it. Per-run structured logs are written by DoctorEngine (run start/found-checks/completion summary) and the startup offline-profile banner is emitted by the StellaOps.Doctor.Startup logger.

The engine resilience hardening (ADR-026 D8) adds the StellaOps.Doctor.Engine meter with the doctor.check.blocking_probe counter (tags check_id / plugin_id), emitted by CheckExecutor when a check ignores its cancellation token past the effective timeout + grace window (see §2 Engine resilience). Hosts that wire AddStellaOpsTelemetry can subscribe to this meter to alert on blocking probes; the same event is also logged at Warning. Discovery timeouts and CanRun failures are surfaced as visible Fail results (and Warning logs) rather than as metrics.

Doctor runs also append deterministic evidence logs through DoctorEvidenceLogWriter, and report metadata is persisted to PostgreSQL (see Report Storage Runtime above).

8) Configuration

The WebService binds the Doctor configuration section to DoctorServiceOptions (Options/DoctorServiceOptions.cs), with the environment-variable prefix DOCTOR_ and optional etc/doctor.yaml / doctor.yaml files (Program.cs).

9) Related Documentation

• Registry Diagnostic Checks
• Registry Compatibility Runbook
• Registry Referrer Troubleshooting