checkId: check.observability.metrics plugin: stellaops.doctor.observability severity: warn tags: [observability, metrics, prometheus, reachability]

Metrics Collection

What It Checks

Resolves a Prometheus health or scrape endpoint from Metrics:Prometheus:Endpoint, Prometheus:Endpoint, Observability:Prometheus:Endpoint, or a local Metrics:Port / Prometheus:Port plus metrics path. The check then probes the endpoint through the injectable IObservabilityReachabilityProbe seam. Explicit Prometheus endpoints default to /-/healthy; local metrics ports probe the configured metrics path.

The check reports skip when metrics are disabled, unconfigured, or no reachability probe is available. It reports warn when the endpoint is configured but unreachable, and pass only when the endpoint answers successfully. Retry attempts are included in evidence for transient probe failures.

Why It Matters

Metrics are the primary input for alerting, SLO tracking, and capacity planning. A present setting is not enough: the scrape or health endpoint must be reachable from the workload that Doctor is validating.

Common Causes

• Metrics were never enabled in the deployment configuration
• The metrics path or port does not match the listener exposed by the service
• A sidecar or reverse proxy blocks local probing
• Prometheus is configured but its health endpoint is not exposed to the Doctor host

How to Fix

Docker Compose

services:
  doctor-web:
    environment:
      Metrics__Enabled: "true"
      Metrics__Path: /metrics
      Metrics__Port: 8080
      Prometheus__Endpoint: http://prometheus:9090

Probe the endpoint from inside the container:

docker compose -f devops/compose/docker-compose.stella-ops.yml exec doctor-web curl -fsS http://prometheus:9090/-/healthy
docker compose -f devops/compose/docker-compose.stella-ops.yml exec doctor-web curl -fsS http://localhost:8080/metrics

Bare Metal / systemd

Bind the metrics port explicitly if the service does not share the main HTTP listener. Make sure the same endpoint is reachable from the service account running Doctor.

Kubernetes / Helm

Align the ServiceMonitor, scrape config, and service port with the same path Doctor probes. If metrics are exposed only through a sidecar, register a local reachability probe that checks the sidecar endpoint.

Verification

stella doctor run --check check.observability.metrics

Related Checks

• check.observability.otel - OpenTelemetry metrics often share the same collector path
• check.observability.alerting - metrics are usually the source for alert rules
• check.observability.logs.shipping - logs are the fallback signal when metrics are missing