Stella Ops Telemetry

Shared OpenTelemetry instrumentation, metric families, redaction, and federation libraries that Stella Ops services consume in-process.

Audience: developers instrumenting Stella Ops services, and operators reasoning about what telemetry the platform emits today versus what remains on the roadmap.

The Telemetry module ships as in-process .NET libraries (StellaOps.Telemetry.Core, StellaOps.Telemetry.Federation, StellaOps.Telemetry.Analyzers) that Stella Ops services consume via dependency injection. It provides:

• OpenTelemetry instrumentation (AddStellaOpsTelemetry) and context propagation;
• log redaction;
• the Stella Ops metric families (golden signals, DORA, NIS2, time-to-first-signal / time-to-evidence, proof coverage/generation, fidelity, P0, unknowns-burndown, triage, attestation/deployment);
• sealed-mode and incident-mode guardrails;
• opt-in federated telemetry (DSSE-signed consent, differential-privacy aggregation, bundle build/verify).

Scope boundary. The deployed observability stack described under Implementation Status below — collectors, Prometheus/Tempo/Loki storage, forensic capture, and the stella telemetry deploy/profile/capture CLI verbs — is roadmap and is not yet implemented in src/Telemetry/. See ./architecture.mdfor the authoritative, code-grounded surface (DI entry points, the full meter table, HTTP endpoints, and config keys).

Responsibilities (current library surface)

• Wire the OpenTelemetry SDK (tracing/metrics/logging) and context propagation into Stella Ops services via AddStellaOpsTelemetry(...), AddTelemetryContextPropagation(), UseStellaOpsTelemetryContext(), and AddTelemetryPropagation() (TelemetryServiceCollectionExtensions).
• Register the Stella Ops metric families through opt-in DI extensions (AddGoldenSignalMetrics, AddDoraMetrics, AddNis2AreaMetrics, AddTimeToFirstSignalMetrics, AddTimeToEvidenceMetrics, AddAttestationMetrics) and enforce metric-label cardinality (MetricLabelGuard / MetricLabelAnalyzer) and credential-logging hygiene (CredentialExceptionLoggingAnalyzer).
• Provide log redaction (AddLogRedaction → LogRedactor / RedactingLogProcessor).
• Enforce sealed-mode egress guardrails (AddSealedModeTelemetry → SealedModeTelemetryService, TelemetryExporterGuard, SealedModeFileExporter) and incident-mode sampling toggles (AddIncidentMode → IncidentModeService).
• Provide opt-in federated telemetry (AddFederatedTelemetry / AddFederatedTelemetrySync): DSSE-signed consent, tenant-bounded k-anonymity and differential-privacy aggregation, bundle build/verify, durable Platform-owned installation state, and certificate-bound peer delivery. The deployed two-installation forcing proof is recorded under docs/qa/feature-checks/runs/telemetry/federated-peer-delivery/run-001/.

Responsibilities (roadmap — not yet implemented)

• Deploy and operate OpenTelemetry collectors for Stella Ops services.
• Provide storage configuration for Prometheus/Tempo/Loki stacks.
• Document smoke tests and offline bootstrapping steps.
• Align metrics and alert packs with module SLOs.

Key components

• Telemetry instrumentation + metric families: src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/.
• Federated telemetry pipeline: src/Telemetry/StellaOps.Telemetry.Federation/.
• Build-time hygiene analyzers (MetricLabelAnalyzer, CredentialExceptionLoggingAnalyzer): src/Telemetry/StellaOps.Telemetry.Analyzers/.
• HTTP endpoints (federation participation/installation administration, NIS2 effectiveness): owned by src/Platform/StellaOps.Platform.WebService/Endpoints/ (FederationTelemetryEndpoints.cs, FederationPeerEndpoints.cs, Nis2TelemetryEndpoints.cs). Current-tenant /participation/* routes use telemetry:participation:read, telemetry:participation:write, or telemetry:facts:write; installation /installation/* routes use platform:federation:read or platform:federation:write. The certificate-authenticated peer receive route is internal and excluded from public OpenAPI. The NIS2 effectiveness route uses analytics.read (see StellaOps.Auth.Abstractions/StellaOpsScopes.cs).
• CLI telemetry surface (shipped): instrumentation under src/Cli/StellaOps.Cli/Telemetry/ (CliMetrics, CliActivitySource, SealedModeTelemetry, TraceparentHttpMessageHandler), the setup-wizard TelemetrySetupStep, and the DORA incident-reporting command (src/Cli/StellaOps.Cli/Commands/DoraIncidentCommandGroup.cs). There is no stella telemetry deploy/profile/capture command group — those are roadmap.
• Roadmap: collector/storage deployment guides (./operations/collector.md, ./operations/storage.md). DevOps collector/dashboard/alert assets live under devops/telemetry/ (collectors, dashboards, grafana, storage, alerts) in the main repo; there is no ops/devops/telemetry/ path.

Integrations & dependencies

• DevOps pipelines for packaging telemetry bundles.
• Module-specific dashboards (scheduler, scanner, etc.).
• Security/Compliance for retention policies.

Operational notes

• DevOps collector configs, dashboards, Grafana provisioning, storage configs, and alert rules live in the main repo under devops/telemetry/ (the older ops/devops/telemetry path does not exist).
• Observability assets: operations/observability.mdand operations/dashboards/telemetry-observability.json (offline import); the TTFS observability dashboard at operations/dashboards/ttfs-observability.json and alerts at operations/alerts/ttfs-alerts.yaml.

Related resources

• Architecture — authoritative code-grounded surface.
• Time-to-first-signal architecture
• Federation architecture
• Collector operations (roadmap)
• Storage operations (roadmap)
• NIS2 effectiveness dashboard
• Federated consent contract v1
• Federated telemetry contract v1
• NIS2 KPI telemetry schema v1
• Federated telemetry operations runbook

Backlog references

• TELEMETRY-OBS-50-001 … 50-004 (Epic 15 collector/storage backlog). The ../../TASKS.md and ops/devops/TASKS.md trackers referenced historically are not present in this repository layout; collector/storage automation remains roadmap (see Implementation Status below).

Implementation Status

The phases below are the original Epic 15 plan. As of 2026-05-29 the deployed collector/storage/forensic stack and the stella telemetry deploy/profile/capture CLI verbs are not yet implemented; the shipped surface is the in-process instrumentation, guardrails, and federation libraries described in ./architecture.md. Phases are retained as roadmap.

Phase 1 – Collector & pipeline profiles (Planned)

• OpenTelemetry collector configs: default, forensic, airgap profiles
• Ingest gateways with TLS/mTLS support
• Attribute redaction policies and tenant isolation
• CLI automation: stella telemetry deploy, stella telemetry profile diff

Phase 2 – Storage backends & retention (Planned)

• Prometheus/Tempo/Loki deployment with retention tiers
• Bucket/object storage with deterministic manifest generation
• Sealed-mode allowlists and offline bundle support
• Remote-write configuration and archivers

Phase 3 – Incident mode & forensic capture (Partial)

The in-process incident-mode service (IncidentModeService, registered via AddIncidentMode) ships today — it toggles enhanced telemetry (sampling up to 100% via IncidentSamplingRate) with JSON-persisted state and TTL, activated through API/CLI/configuration. Forensic OTLP archive bundles and Notify escalation hooks remain roadmap.

• Incident toggles via CLI/API for sampling adjustments — implemented (IncidentModeService)
• Tail sampling to 100% during incidents — incident-mode sampling override implemented; collector-side tail sampling roadmap
• Forensic bundle generation: OTLP archives with manifest/signature — roadmap
• Notify hooks for incident escalation — roadmap

Phase 4 – Observability dashboards & automation (Planned)

• Service SLO dashboards: queue depth, policy latency, ingestion violations
• Alert rules: burn-rate, collector failure, exporter backlog
• Grafana packages for core services
• Self-observability metrics

Phase 5 – Offline & compliance (Planned)

• Offline Kit artifacts: collector binaries/configs, import scripts
• Deterministic bundles with signed manifests
• Replay tooling and compliance checklists
• File-based exporters for air-gapped environments

Phase 6 – Hardening & SOC handoff (Planned)

• RBAC integration and audit logging
• Incident response runbooks and performance tuning
• Integration tests across services
• SOC handoff package with control objectives

Key Acceptance Criteria

• Collectors ingest metrics/logs/traces with redaction rules and tenant isolation
• Storage backends retain data per SLAs with deterministic manifests
• Incident mode triggers forensic capture with signed bundles
• Dashboards/alerts cover service SLOs and telemetry stack health
• CLI automates config rollout, forensic capture, verification
• Offline bundles replay telemetry in sealed environments

Technical Decisions & Risks

• PII leakage prevented via strict redaction processors, policy-managed allowlists
• Collector overload managed with horizontal scaling, batching, circuit breakers
• Storage cost controlled via tiered retention, compression, pruning, offline archiving
• Air-gap drift mitigated with offline kit refresh schedule, manifest verification
• Alert fatigue reduced with burn-rate alerts, deduping, SOC runbooks

Operational Assets (Sprint 0330 · 2025-11-30)

• Observability runbook: operations/observability.md
• Dashboard placeholder: operations/dashboards/telemetry-observability.json
• NIS2 effectiveness dashboard: operations/dashboards/nis2-effectiveness-dashboard.json (doc: operations/nis2-effectiveness-dashboard.md)
• TTFS observability dashboard + alerts: operations/dashboards/ttfs-observability.json, operations/alerts/ttfs-alerts.yaml

Epic alignment

• Epic 15 – Observability & Forensics: deliver collector/storage deployments, forensic evidence retention, and observability bundles with deterministic configuration.