Stella Ops Doctor

Self-service diagnostics for Stella Ops deployments

Overview

Doctor has two read surfaces: the CLI’s local plugin engine and the installed services’ doctor-check/v1 endpoints. Owned-state storage checks belong to the service endpoints and Platform aggregate; stella doctor list does not enumerate them. See how to read owned-state checks.

The Doctor system provides comprehensive diagnostics for Stella Ops deployments, enabling operators, DevOps engineers, and developers to:

Diagnose what is working and what is not
Understand why failures occur with collected evidence
Remediate issues with copy/paste commands
Verify fixes with re-runnable checks

Quick Start

CLI

# Quick health check
stella doctor

# Full diagnostic with all checks
stella doctor --mode full

# Check specific category
stella doctor --category database

# Export report for support
stella doctor export --output diagnostic-bundle.zip

# Apply safe fixes from a report (dry-run by default)
stella doctor fix --from doctor-report.json --apply

UI

Navigate to /ops/doctor in the Stella Ops console to access the interactive Doctor Dashboard. Remediation follows the configured execution gates in ADR-026 and is disabled until its required opt-in and authorization conditions are met.

Installed service checks

Use the protected HTTP read surfaces for installed service state:

GET /doctor/{service}/checks
GET /api/v1/platform/doctor/capabilities
GET /api/v1/platform/doctor/aggregate

The first returns live service results; the capability catalog records claims; the aggregate pulls registered services and reports individual failures. All three reads require ops.health. The standalone Doctor WebService and its old /api/v1/doctor/* host routes are retired; the retained source host is used for contract coverage. Scheduled estate checks belong to JobEngine estate packs. See the current Doctor architecture.

Discover the available checks

Plugin availability depends on the local CLI configuration and loaded plugins; use stella doctor list for that catalog. Installed service checks, including Evidence Timestamp Assurance and owned-state measurements, are discovered through their service endpoints and Platform catalog. There is no fixed global check count.

Setup Wizard Essential Checks

The following checks are mandatory for the setup wizard to validate a new installation:

DB connectivity + schema version (stellaops.doctor.database)
- check.db.connection - Database is reachable
- check.db.schema.version - Schema version matches expected
Attestation store availability (stellaops.doctor.attestation)
- check.attestation.rekor.connectivity - Rekor transparency log reachable
- check.attestation.cosign.keymaterial - Signing keys available (file/KMS/keyless)
- check.attestation.clock.skew - System clock synchronized (<5s skew)
Artifact verification pipeline (stellaops.doctor.verification)
- check.verification.artifact.pull - Test artifact accessible by digest
- check.verification.signature - DSSE signatures verifiable
- check.verification.sbom.validation - SBOM (CycloneDX/SPDX) valid
- check.verification.vex.validation - VEX document valid
- check.verification.policy.engine - Policy evaluation passes

Check ID Convention

check.{category}.{subcategory}.{specific}

Examples:

check.config.required
check.database.migrations.pending
check.services.gateway.routing
check.integration.scm.github.auth
check.attestation.rekor.connectivity
check.verification.sbom.validation

CLI Reference

See CLI Reference for complete command documentation.

Common Commands

# Quick health check (tagged 'quick' checks only)
stella doctor --quick

# Full diagnostic with all checks
stella doctor --full

# Filter by category
stella doctor --category database
stella doctor --category security

# Filter by plugin
stella doctor --plugin scm.github

# Run single check
stella doctor --check check.database.migrations.pending

# Output formats
stella doctor --format json
stella doctor --format markdown
stella doctor --format text

# Filter output by severity
stella doctor --severity fail,warn

# Export diagnostic bundle
stella doctor export --output diagnostic.zip
stella doctor export --include-logs --log-duration 4h

Exit Codes

Code	Meaning
0	All checks passed
1	One or more warnings
2	One or more failures
3	Doctor engine error
4	Invalid arguments
5	Timeout exceeded

Output Example

Stella Ops Doctor
=================

Running 47 checks across 8 plugins...

[PASS] check.config.required
  All required configuration values are present

[PASS] check.database.connectivity
  PostgreSQL connection successful (latency: 12ms)

[WARN] check.tls.certificates.expiry
  Diagnosis: TLS certificate expires in 14 days

  Evidence:
    Certificate: /etc/ssl/certs/stellaops.crt
    Subject: CN=stellaops.example.com
    Expires: 2026-01-26T00:00:00Z
    Days remaining: 14

  Likely Causes:
    1. Certificate renewal not scheduled
    2. ACME/Let's Encrypt automation not configured

  Fix Steps:
    # 1. Check current certificate
    openssl x509 -in /etc/ssl/certs/stellaops.crt -noout -dates

    # 2. Renew certificate (if using certbot)
    sudo certbot renew --cert-name stellaops.example.com

    # 3. Restart services to pick up new certificate
    sudo systemctl restart stellaops-gateway

  Runbook:
    https://docs.stella-ops.org/runbooks/tls-certificate-renewal

  Verification:
    stella doctor --check check.tls.certificates.expiry

[FAIL] check.database.migrations.pending
  Diagnosis: 3 pending release migrations detected in schema 'auth'

  Evidence:
    Schema: auth
    Current version: 099_add_dpop_thumbprints
    Pending migrations:
      - 100_add_tenant_quotas
      - 101_add_audit_retention
      - 102_add_session_revocation

  Likely Causes:
    1. Release migrations not applied before deployment
    2. Migration files added after last deployment

  Fix Steps:
    # 1. Backup database first (RECOMMENDED)
    pg_dump -h localhost -U stella_admin -d stellaops -F c \
      -f stellaops_backup_$(date +%Y%m%d_%H%M%S).dump

    # 2. Restart the owning service; it converges its own schema at startup
    docker compose up -d --force-recreate authority

    # 3. Verify migrations applied
    docker compose logs --since 5m authority | grep 'Migration:'

  Verification:
    stella doctor --check check.database.migrations.pending

--------------------------------------------------------------------------------
Summary: 44 passed, 2 warnings, 1 failed (47 total)
Duration: 8.3s
--------------------------------------------------------------------------------

Export Bundle

The Doctor export feature creates a diagnostic bundle for support escalation:

stella doctor export --output diagnostic-bundle.zip

The bundle contains:

doctor-report.json - Full diagnostic report
doctor-report.md - Human-readable report
environment.json - Environment information
system-info.json - System details (OS, runtime, memory)
config-sanitized.json - Sanitized configuration (secrets redacted)
logs/ - Recent log files (optional)
README.md - Bundle contents guide

Export Options

# Include logs from last 4 hours
stella doctor export --include-logs --log-duration 4h

# Exclude configuration
stella doctor export --no-config

# Custom output path
stella doctor export --output /tmp/support-bundle.zip

Security

Secret Redaction

All evidence output is sanitized. Sensitive values (passwords, tokens, connection strings) are replaced with ***REDACTED*** in:

Console output
JSON exports
Diagnostic bundles
Log files

RBAC Permissions

Scope	Description
`doctor:run`	Execute doctor checks
`doctor:run:full`	Execute all checks including sensitive
`doctor:export`	Export diagnostic reports
`admin:system`	Access system-level checks

Plugin Development

To create a custom Doctor plugin, implement IDoctorPlugin:

public class MyCustomPlugin : IDoctorPlugin
{
    public string PluginId => "stellaops.doctor.custom";
    public string DisplayName => "Custom Checks";
    public Version Version => new(1, 0, 0);
    public DoctorCategory Category => DoctorCategory.Integration;

    public bool IsAvailable(IServiceProvider services) => true;

    public IReadOnlyList<IDoctorCheck> GetChecks(DoctorPluginContext context)
    {
        return new IDoctorCheck[]
        {
            new MyCustomCheck()
        };
    }

    public Task InitializeAsync(DoctorPluginContext context, CancellationToken ct)
        => Task.CompletedTask;
}

Implement checks using IDoctorCheck:

public class MyCustomCheck : IDoctorCheck
{
    public string CheckId => "check.custom.mycheck";
    public string Name => "My Custom Check";
    public string Description => "Validates custom configuration";
    public DoctorSeverity DefaultSeverity => DoctorSeverity.Fail;
    public IReadOnlyList<string> Tags => new[] { "custom", "quick" };
    public TimeSpan EstimatedDuration => TimeSpan.FromSeconds(2);

    public bool CanRun(DoctorPluginContext context) => true;

    public async Task<DoctorCheckResult> RunAsync(
        DoctorPluginContext context,
        CancellationToken ct)
    {
        // Perform check logic
        var isValid = await ValidateAsync(ct);

        if (isValid)
        {
            return DoctorCheckResult.Pass(
                checkId: CheckId,
                diagnosis: "Custom configuration is valid",
                evidence: new Evidence
                {
                    Description = "Validation passed",
                    Data = new Dictionary<string, string>
                    {
                        ["validated_at"] = context.TimeProvider.GetUtcNow().ToString("O")
                    }
                });
        }

        return DoctorCheckResult.Fail(
            checkId: CheckId,
            diagnosis: "Custom configuration is invalid",
            evidence: new Evidence
            {
                Description = "Validation failed",
                Data = new Dictionary<string, string>
                {
                    ["error"] = "Configuration file missing"
                }
            },
            remediation: new Remediation
            {
                Steps = new[]
                {
                    new RemediationStep
                    {
                        Order = 1,
                        Description = "Create configuration file",
                        Command = "cp /etc/stellaops/custom.yaml.sample /etc/stellaops/custom.yaml",
                        CommandType = CommandType.Shell
                    }
                }
            });
    }
}

services.AddSingleton<IDoctorPlugin, MyCustomPlugin>();

Architecture

+------------------+     +------------------+     +------------------+
|       CLI        |     |        UI        |     |    External      |
|  stella doctor   |     |   /ops/doctor    |     |   Monitoring     |
+--------+---------+     +--------+---------+     +--------+---------+
         |                        |                        |
         v                        v                        v
+------------------------------------------------------------------------+
|                         Doctor API Layer                                |
|  POST /api/v1/doctor/run    GET /api/v1/doctor/checks                  |
|  GET /api/v1/doctor/report  WebSocket /api/v1/doctor/stream            |
+------------------------------------------------------------------------+
         |
         v
+------------------------------------------------------------------------+
|                      Doctor Engine (Core)                               |
|  +------------------+  +------------------+  +------------------+       |
|  | Check Registry   |  | Check Executor   |  | Report Generator |       |
|  | - Discovery      |  | - Parallel exec  |  | - JSON/MD/Text   |       |
|  | - Filtering      |  | - Timeout mgmt   |  | - Remediation    |       |
|  +------------------+  +------------------+  +------------------+       |
+------------------------------------------------------------------------+
         |
         v
+------------------------------------------------------------------------+
|                        Plugin System                                    |
+--------+---------+---------+---------+---------+---------+-------------+
         |         |         |         |         |         |
         v         v         v         v         v         v
+--------+  +------+  +------+  +------+  +------+  +------+  +----------+
| Core   |  | DB & |  |Service|  | SCM  |  |Regis-|  |Observ-|  |Security |
| Plugin |  |Migra-|  | Graph |  |Plugin|  | try  |  |ability|  | Plugin  |
|        |  | tions|  |Plugin |  |      |  |Plugin|  |Plugin |  |         |
+--------+  +------+  +------+  +------+  +------+  +------+  +----------+

CLI Reference - Complete CLI command reference
Doctor Capabilities Specification - Full technical specification
Plugins Reference - Built-in plugins, their checks, and configuration
Evidence Schemas - Standardized evidence fields per check
Plugin Development - Writing your own IDoctorPlugin

Troubleshooting

Doctor Engine Error (Exit Code 3)

If stella doctor returns exit code 3:

Check the error message for details
Verify required services are running
Check connectivity to databases
Review logs at /var/log/stellaops/doctor.log

Timeout Exceeded (Exit Code 5)

If checks are timing out:

# Increase per-check timeout
stella doctor --timeout 60s

# Run with reduced parallelism
stella doctor --parallel 2

Checks Not Found

If expected checks are not appearing:

Verify plugin is registered in DI
Check CanRun() returns true for your environment
Review plugin initialization logs