Telemetry Observability Runbook

This runbook tells operators how to observe the Stella Ops telemetry pipeline itself: which dashboards and metrics to watch, the health and alert signals that matter, and how to verify the pipeline offline. For deployment, see the collector and storage runbooks; for the design, see the telemetry architecture.

Audience: platform operators and on-call engineers monitoring collector health, ingestion, retention, and query latency — including air-gapped deployments.

Dashboards (offline import)

Grafana JSON: docs/modules/telemetry/operations/dashboards/telemetry-observability.json (import locally; no external data sources assumed).
Planned panels: collector uptime, scrape errors, ingestion/backlog per tenant, storage retention headroom, query latency p95/p99, and OTLP export errors.

Key metrics

telemetry_collector_uptime_seconds — per-collector uptime.
telemetry_scrape_failures_total{job} — scrape failures per job.
telemetry_ingest_backlog — queued spans/logs/metrics awaiting storage.
telemetry_storage_retention_percent_used — storage utilization against retention budget.
telemetry_query_latency_seconds_bucket{route} — API/query latency.
telemetry_otlp_export_failures_total{signal} — OTLP export failures by signal.

Logs & traces

Correlate by trace_id and tenant; include collector_id, pipeline, exporter fields.
Traces disabled by default for air-gap; enable by setting OTLP endpoints to on-prem collectors.

Health/diagnostics

/health/liveness and /health/readiness (collector + storage gateway) check exporter reachability and disk headroom.
/status exposes build version, commit, feature flags; verify against offline bundle manifest.
Storage probe: GET /api/storage/usage (if available) to confirm retention headroom; otherwise rely on Prometheus metrics.

Alert hints

OTLP export failures > 0 over 5m.
Ingest backlog above threshold (configurable per tenant/workload).
Query latency p99 > 1s for /api/query routes.
Storage utilization > 85% of retention budget.

Offline verification steps

Import the Grafana JSON locally and point it at the Prometheus scrape labeled telemetry.
Run a collector smoke test: push sample OTLP spans, logs, and metrics to the local collector and confirm the metrics above appear in Prometheus.
Fetch /status and compare commit/version against the Offline Kit bundle manifest.

Module overview: telemetry README.
Design: telemetry architecture and implementation plan.
Dashboard source: dashboards/telemetry-observability.json.