checkId: check.metrics.prometheus.scrape plugin: stellaops.doctor.observability severity: warn tags: [observability, metrics, prometheus, air-gap]

Prometheus Scrape

This Stella Ops Doctor diagnostic proves that the platform’s metrics endpoint is live and serving Prometheus-formatted samples, so that alerting rules can fire and dashboards stay populated. It performs a live HTTP probe and is air-gap aware (see below).

What It Checks

Verifies that the application metrics endpoint is accessible for Prometheus scraping. The check:

• Reads Metrics:Path (default /metrics), Metrics:Port (default 8080), and Metrics:Host (default localhost).
• Sends a GET request to http://{host}:{port}{path} with a 5-second timeout.
• Counts the number of Prometheus-formatted metric lines in the response.
• Passes if the endpoint returns a successful response with metrics.
• Warns on non-success status codes, timeouts, or connection failures.

The check only runs when Metrics:Enabled is set to true.

Air-Gap Behavior

When the deployment is sealed by AirGap policy, or STELLAOPS_DOCTOR_OFFLINE_PROFILE=true forces Doctor offline mode, external Prometheus metrics endpoints return Skip with airGapSealed=true. Loopback, link-local, RFC1918, and unique-local IPv6 endpoints are still probed because Prometheus scraping is commonly internal to sealed deployments.

If your sealed network uses a routable non-private address for an internal metrics endpoint, Doctor cannot distinguish it from external egress without DNS probing. Prefer an RFC1918, loopback, link-local, or unique-local endpoint for this check.

Why It Matters

Prometheus metrics provide real-time visibility into service health, request latencies, error rates, and resource utilization. Without a scrapeable metrics endpoint, alerting rules cannot fire, dashboards go blank, and capacity planning has no data.

Common Causes

• Metrics endpoint not enabled in configuration
• Wrong port configured
• Service not running on the expected port
• Authentication required but not configured for Prometheus
• Firewall blocking the metrics port

How to Fix

Docker Compose

environment:
  Metrics__Enabled: "true"
  Metrics__Path: "/metrics"
  Metrics__Port: "8080"

# Test metrics endpoint
docker exec <platform-container> curl -s http://localhost:8080/metrics | head -5

Bare Metal / systemd

Edit appsettings.json:

{
  "Metrics": {
    "Enabled": true,
    "Path": "/metrics",
    "Port": 8080
  }
}

# Verify metrics are exposed
curl -s http://localhost:8080/metrics | head -5

# Check port binding
netstat -an | grep 8080

Kubernetes / Helm

metrics:
  enabled: true
  port: 8080
  path: "/metrics"
  serviceMonitor:
    enabled: true

Add Prometheus annotations to the pod:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"
  prometheus.io/path: "/metrics"

Verification

stella doctor run --check check.metrics.prometheus.scrape

Related Checks

• check.telemetry.otlp.endpoint — verifies OTLP collector endpoint reachability
• check.logs.directory.writable — verifies log directory is writable