Telemetry Architecture

Audience: developers instrumenting Stella Ops services, and operators reasoning about what telemetry the platform emits today versus what remains on the roadmap. For the module overview and quick links, see ./README.md.

Telemetry is the shared observability foundation for the Stella Ops release control plane: OpenTelemetry instrumentation, the platform metric families, log redaction, sealed/incident-mode guardrails, and opt-in federated telemetry — all consumed in-process by other modules.

Aligned with Epic 15 (Observability & Forensics). Implementation status (2026-05-29): src/Telemetry/ ships as a set of in-process .NET libraries that Stella Ops services consume via dependency injection — not a standalone collector/storage deployment. This document describes the implemented library surface first, then records the still-unbuilt Epic 15 collector/storage roadmap as explicitly marked forward specs.

0) What is implemented vs. roadmap

The Telemetry module today consists of three libraries plus consuming endpoints owned by other modules:

Roadmap (NOT YET IMPLEMENTED). A deployed OpenTelemetry Collector tier, Prometheus/Tempo/Loki storage backends, collector config profiles (default/forensic/airgap), forensic OTLP archive bundles, and stella telemetry CLI verbs are tracked in README.md (Phases 1-6) and remain planned. They are described in section 7 below and are clearly tagged so readers do not mistake them for shipped behaviour.

1) Service-side instrumentation (AddStellaOpsTelemetry)

The core entry point is TelemetryServiceCollectionExtensions.AddStellaOpsTelemetry(...), which wires the OpenTelemetry SDK (tracing, metrics, logging) for a host service and returns the OpenTelemetryBuilder for further chaining. Behaviour:

• Binds options from the Telemetry configuration section into StellaOpsTelemetryOptions (sub-sections: Collector, Propagation, Labels).
• Registers TelemetryServiceDescriptor, TelemetryExporterGuard, ITelemetryContextAccessor, MetricLabelGuard, and a TelemetryPropagationHandler.
• Default tracing instrumentation: ASP.NET Core + HttpClient. Default metrics instrumentation: runtime + the StellaOps.Telemetry meter.
• Honours OTEL_SDK_DISABLED / Telemetry:Enabled=false to short-circuit SDK setup (used by WebApplicationFactory tests).

Additional opt-in DI extensions (all in TelemetryServiceCollectionExtensions):

• AddTelemetryContextPropagation() registers TelemetryContextAccessor plus gRPC server/client interceptors.
• AddLogRedaction(...) registers ILogRedactor / LogRedactor driven by LogRedactionOptions.
• AddGoldenSignalMetrics(...), AddTimeToEvidenceMetrics(...), AddTimeToFirstSignalMetrics(...), AddDoraMetrics(...), AddNis2AreaMetrics(...), AddAttestationMetrics() register the metric families.
• AddIncidentMode(...) and AddSealedModeTelemetry(...) register the incident/sealed-mode services.
• UseStellaOpsTelemetryContext() (ASP.NET Core middleware) and AddTelemetryPropagation() (HttpClient handler) propagate context across the request pipeline.

Collector exporter (OTLP target, not a deployed tier)

StellaOpsTelemetryOptions.CollectorOptions configures the OTLP exporter that ships traces/metrics out of the process. It is not a deployed OpenTelemetry Collector. Keys:

When an IEgressPolicy is registered, the exporter’s HttpClientFactory routes through EgressHttpClientFactory so air-gap egress rules apply to telemetry export.

Propagation context

TelemetryContext carries CorrelationId/TraceId, TenantId, Actor, and ImposedRule across HTTP, gRPC, background jobs, and CLI. PropagationOptions configures the header names (X-StellaOps-TenantId, x-stella-actor, x-stella-imposed-rule, X-StellaOps-TraceId). Cardinality is bounded by MetricLabelGuard (Labels:MaxDistinctValuesPerLabel default 50, Labels:MaxLabelLength default 64).

2) Metric families (StellaOps.Telemetry.Core)

Each family creates its own System.Diagnostics.Metrics.Meter; there is no shared Prometheus/Tempo/Loki storage in this module. Emission is via the OTLP exporter configured above (or the sealed-mode file exporter when offline). Implemented meters:

3) Sealed mode, incident mode, and redaction (guardrails)

• Redaction. LogRedactor + RedactingLogProcessor strip configured keys (LogRedactionOptions). The CredentialExceptionLoggingAnalyzer Roslyn analyzer additionally fails the build when exception logging would leak credentials. There is no policy-server-managed allow-list profile in v1; redaction keys are local options.
• Sealed-mode guard. SealedModeTelemetryService derives IsSealed from IEgressPolicy.IsSealed (falling back to Telemetry:Sealed:Enabled). When sealed, IsExternalExportAllowed(endpoint) returns false and RecordDriftEvent logs blocked exports; TelemetryExporterGuard short-circuits OTLP export. Sampling is clamped to Telemetry:Sealed:MaxSamplingPercent (default 10%). The sealed-mode OTLP file exporter (SealedModeFileExporter) writes to Telemetry:Sealed:FilePath (default ./logs/telemetry-sealed.otlp) with rotation (MaxBytes 10 MB, MaxRotatedFiles 3) and Unix 0600 permissions.
• Incident mode. IncidentModeService toggles enhanced telemetry (sampling up to 100% via IncidentSamplingRate, faster flush). It is an in-process service activated via API/CLI/configuration source, not an Orchestrator-driven promotion. State is persisted to a local JSON file (Telemetry:Incident:StateFilePath, default ~/.stellaops/incident-mode.json) and restored on startup; TTL defaults to 30 min (MaxTtl 24 h) with bounded extensions. In sealed mode, incident mode may override the sampling ceiling only if Telemetry:Sealed:AllowIncidentModeOverride is set (default true); Telemetry:Incident:DisableInSealedMode (default true) otherwise blocks activation. Activation/deactivation emit structured audit log lines (telemetry.incident.activated/deactivated/expired).

4) APIs & integration

The Telemetry libraries expose no HTTP surface of their own. Consuming HTTP endpoints live in StellaOps.Platform.WebService.

Federated telemetry — FederationTelemetryEndpoints.MapFederationTelemetryEndpoints, base path /api/v1/telemetry/federation. /participation/* is current-tenant state and requires tenant participation scopes; /installation/* is Platform installation state and requires installation-operator federation scopes without manufacturing tenant context.

NIS2 effectiveness — Nis2TelemetryEndpoints.MapNis2TelemetryEndpoints:

Platform persists consent, eligible facts, installation privacy accounting, consent-set commitments, bundles, peer delivery receipts, and imported aggregate intelligence in PostgreSQL. Local Synced means durable outbox acceptance. When the optional peer lane is enabled, only a destination-bound, mTLS-authenticated HTTPS response carrying the deterministic signed-bundle receipt changes an outbox row to delivered.

CLI integration today is limited to: telemetry instrumentation of the CLI itself (src/Cli/StellaOps.Cli/Telemetry/ — CliMetrics, CliActivitySource, SealedModeTelemetry, TraceparentHttpMessageHandler), a setup-wizard TelemetrySetupStep, and the DoraIncidentCommandGroup (DORA incident reporting). There is no stella telemetry deploy/capture/profile command group; those remain roadmap (see section 7).

5) Persistence

Telemetry ships as in-process libraries and owns no standalone PostgreSQL schema. Platform migration 007 owns installation identity, consent, facts, privacy, bundle, and local outbox state. Migration 008 adds durable imported aggregate intelligence; migration 009 adds leased outbox claims, peer acknowledgements, and digest-bound inbox receipts. The current-tenant /participation/facts route supplies authenticated durable ingress. Sealed mode skips both aggregation and peer delivery.

6) Federation pipeline (StellaOps.Telemetry.Federation)

FederationServiceCollectionExtensions.AddFederatedTelemetry(...) registers the federation services; AddFederatedTelemetrySync() adds the FederatedTelemetrySyncService background worker. Pipeline:

1. Consent (ConsentManager) — grant/revoke/verify per tenant. Grants produce a DSSE-signed ConsentProof (predicate stella.ops/federatedConsent@v1); Platform’s production store persists proofs and retained revocations.
2. Aggregation (TelemetryAggregator) — groups TelemetryFacts by CVE, bounds each consenting tenant’s contribution, suppresses buckets below KAnonymityThreshold distinct tenants, adds cryptographically sampled Laplacian noise sized from the contribution sensitivity, and atomically spends from the installation privacy ledger.
3. Privacy budget (PostgresFederationPrivacyBudgetTracker in Platform) — persists installation epsilon spend against EpsilonBudget and resets it by configured period without concurrent overspend.
4. Bundle build/verify (FederatedTelemetryBundleBuilder) — canonicalises non-suppressed buckets, derives a deterministic bundle GUID from a SHA-256 seed, and DSSE-signs the payload (predicate stella.ops/federatedTelemetry@v1). Verification re-checks the envelope digest, payload type, signer key, consent-set commitment, aggregate totals, privacy spend, timestamps, and the complete public bucket projection before any peer intelligence is accepted.
5. Synchronization boundary (FederatedTelemetrySyncService) — disabled by default. Enabled production startup fails with federation.startup.durable_prerequisites_unavailable if a process-local fallback store/outbox remains selected. A cycle atomically claims eligible facts, builds and persists a signed installation bundle, and enqueues a pending local outbox row. It does not claim peer delivery.

Platform binds FederatedTelemetryOptions exclusively from Platform:TelemetryFederation, not Platform:Federation (the latter belongs to ReleaseOrchestrator regional federation). Options include Enabled (default false), KAnonymityThreshold, EpsilonBudget, BudgetResetPeriod, AggregationInterval, SealedModeEnabled, SiteId, predicate types, and DSSE material. DSSE security posture is detailed in section 9.

7) Roadmap — collector, storage, and forensic capture (NOT YET IMPLEMENTED)

The following Epic 15 capabilities are specified in README.md (Phases 1-6) but are not present in src/Telemetry/as of 2026-05-29. They are retained here as forward specs:

• Collector tier. Deployed OpenTelemetry Collector instances per environment (ingest TLS/mTLS, OTLP receivers, tail-based sampling) with config profiles default, forensic (high-retention), airgap (file-based exporters), delivered via Offline Kit.
• Storage backends. Prometheus (metrics, remote-write, retention windows — default 30 days, forensic 180 days), Tempo/Jaeger (traces, object/filesystem block storage with deterministic chunk manifests), Loki (logs in immutable chunks, hashed index shards).
• Forensic archive. Periodic export of raw OTLP records into signed bundles (otlp/metrics.pb, otlp/traces.pb, otlp/logs.pb, manifest.json) for compliance replay.
• Roadmap HTTP/CLI surface. GET /telemetry/config/profile/{name}, POST /telemetry/incidents/mode, GET /telemetry/exports/forensic/{window}, and stella telemetry deploy/capture/profile diff are not implemented; the implemented incident toggle and federation/NIS2 endpoints in sections 3-4 are the current reality.
• Roadmap meta-metrics. collector_export_failures_total, telemetry_bundle_generation_seconds, telemetry_incident_mode{state} are illustrative target names and do not exist in source today; the implemented self-observability metrics are the sealed-mode counters under StellaOps.Telemetry.SealedMode (sections 2-3).
• Offline collector packaging. Offline Kit collector binaries/config, bootstrap scripts, dashboards, alert rules, and OTLP replay tooling. Today the only offline telemetry path is the sealed-mode OTLP file exporter (section 3) and the Grafana dashboard JSON inventoried under operations/dashboards/.

8) Observability of the telemetry stack (implemented)

• Self-observability is emitted through the meters in section 2; the sealed-mode counters (stellaops.telemetry.sealed.seal_events, stellaops.telemetry.sealed.unseal_events, stellaops.telemetry.sealed.drift_events, stellaops.telemetry.sealed.blocked_exports) are the concrete signals for guardrail health.
• Health endpoints/dashboards for a deployed collector/storage cluster are part of the section 7 roadmap.

9) DORA Metrics

Stella Ops tracks the four key DORA (DevOps Research and Assessment) metrics for software delivery performance:

9.1) Metrics Tracked

• Deployment Frequency (dora_deployments_total, dora_deployment_frequency_per_day) — How often deployments occur per day/week.
• Lead Time for Changes (dora_lead_time_hours) — Time from commit to deployment in production.
• Change Failure Rate (dora_deployment_failure_total, dora_change_failure_rate_percent) — Percentage of deployments requiring rollback, hotfix, or failing.
• Mean Time to Recovery (MTTR) (dora_time_to_recovery_hours) — Average time to recover from incidents.

Implementation note: metric names above are emitted by DoraMetrics (meter StellaOps.DORA) via instruments including dora_deployments_total, dora_deployment_duration_seconds, dora_lead_time_hours, dora_deployment_success_total, dora_deployment_failure_total, dora_incidents_total, dora_incidents_resolved_total, dora_time_to_recovery_hours, and dora_slo_breach_total.

9.2) Performance Classification

DoraMetrics.ClassifyPerformance(deploymentFrequencyPerDay, leadTimeHours, changeFailureRatePercent, mttrHours) classifies teams into DoraPerformanceLevel (Elite=4, High=3, Medium=2, Low=1, Unknown=0). The thresholds below reflect the implemented conditional (the in-code comment + DoraPerformanceLevel.Elite XML doc were reconciled to <24h on 2026-07-12, CLO-7):

• Elite: >= 1 deploy/day, < 24h lead time, < 15% CFR, < 1h MTTR.
• High: >= ~0.14 deploy/day (≈1/week), < 168h (1 week) lead time, <= 30% CFR, < 24h MTTR.
• Medium: >= ~0.033 deploy/day (≈1/month), < 4320h (≈6 months) lead time, <= 45% CFR, < 168h (1 week) MTTR.
• Low: any remaining team with deploymentFrequencyPerDay > 0; Unknown when there is no deployment activity.

9.3) Integration Points

• IDoraMetricsService — Service interface for recording deployments and incidents (default impl InMemoryDoraMetricsService).
• DoraMetrics — OpenTelemetry-style metrics class with SLO breach tracking.
• DI registration: services.AddDoraMetrics(options => { ... }).
• Events are recorded when Release Orchestrator completes promotions or rollbacks.

9.4) SLO Tracking

Configurable SLO targets via DoraMetricsOptions:

• LeadTimeSloHours (default: 24)
• DeploymentFrequencySloPerDay (default: 1)
• ChangeFailureRateSloPercent (default: 15)
• MttrSloHours (default: 1)

SLO breaches are recorded as dora_slo_breach_total with metric label.

9.5) Outcome Analytics and Attribution (Sprint 20260208_065)

Telemetry now includes deterministic executive outcome attribution built on top of the existing DORA event stream:

• IOutcomeAnalyticsService (src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/IOutcomeAnalyticsService.cs)
• DoraOutcomeAnalyticsService (src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/DoraOutcomeAnalyticsService.cs)
• Outcome report models (src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/OutcomeAnalyticsModels.cs)

Outcome attribution behavior:

• Produces OutcomeExecutiveReport for a fixed tenant/environment/time window with deterministic ordering.
• Adds MTTA support via DoraIncidentEvent.AcknowledgedAt and TimeToAcknowledge.
• Groups deployment outcomes by normalized pipeline (pipeline-a, pipeline-b, unknown) with per-pipeline change failure rate and median lead time.
• Groups incidents by severity with resolved/acknowledged counts plus MTTA/MTTR aggregates.
• Produces daily cohort slices across the requested date range for executive trend views.

Dependency injection integration:

• TelemetryServiceCollectionExtensions.AddDoraMetrics(...) also registers IOutcomeAnalyticsService (and IDoraIncidentClassifier), so existing telemetry entry points automatically expose attribution reporting without additional module wiring.

Verification coverage:

• src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core.Tests/OutcomeAnalyticsServiceTests.cs
• src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core.Tests/DoraMetricsServiceTests.cs

9.6) DORA ICT Incident Classification (Sprint 20260430_111)

Telemetry Core exposes a deterministic DORA ICT incident classifier:

• IDoraIncidentClassifier and DoraIncidentClassifier
• Classifier input models: DoraIncidentFacts, DoraTenantComplianceProfileSnapshot, and DoraIncidentTelemetrySnapshot
• Result model: DoraIncidentClassificationResult, pinned to stella.ops/doraIncidentClassification@v1 and EU 2024/1772

Classifier behavior:

• Computes all seven DORA criteria from supplied incident facts, tenant compliance profile thresholds, and frozen telemetry snapshots.
• Does not embed live regulatory thresholds or fetch external data. Tenant/operator threshold values are explicit nullable inputs.
• Emits DoraCriterionAssessmentStatus.InsufficientData with an insufficient-data rationale and criterion-local MissingInputs whenever facts or thresholds are incomplete.
• Uses deterministic normalization for evidence refs, jurisdictions, currencies, critical service IDs, and override ordering.
• Supports v1 operator overrides for thresholdMet. Each override updates the criterion result and emits a separate DoraClassificationOverrideAuditEvent containing criterion, field, before/after value, reason, actor, timestamp, and evidence ref. Telemetry Core returns this event for downstream durable ledger/audit persistence; it does not own a durable audit store.

Verification coverage:

• src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core.Tests/DoraIncidentClassifierTests.cs

10) NIS2 Area Metrics (Sprint 20260430_060)

Telemetry Core exposes the NIS2 thirteen-area KPI metric family for effectiveness reporting:

• INis2AreaMetricsService records KPI samples, emits observable gauges, and builds deterministic area reports.
• Nis2AreaMetricCatalog pins the 31 default KPI definitions from docs/europe/nis2-kpi-targets-defaults.yaml.
• Metric names follow nis2_area_<area>_<metric>_<unit> with area numbers 1 through 13, matching docs/europe/nis2-kpi-targets-defaults.yaml.
• Metric labels are limited to tenant, control_id, and severity; evidence refs stay out of labels to avoid PII and cardinality drift.
• Areas 8, 10, and 13 are tenant-supplied in v1. Stella records opaque evidence refs only and does not become a customer workforce, HR, or physical-security source of record.
• Platform WebService registers AddNis2AreaMetrics() and exposes the Web-facing read model at GET /api/telemetry/nis2/effectiveness?tenantId=<tenantId>&window=<rolling-30d|rolling-90d>.
• The Platform route maps INis2AreaMetricsService.GetAreaReportAsync into nis2-effectiveness-dashboard-v1, preserving 13 ordered areas, catalog metric targets, latest seeded measurements, linked controls/evidence refs, and nullable target/control snapshot hashes until the downstream N1 snapshot sources are live.
• Nis2AreaMetricsService.RecordMeasurementAsync classifies each live KPI sample against the catalog effective target bands and emits Nis2EffectivenessThresholdBreach only when a metric first enters warning/critical or escalates from warning to critical. An OK sample resets the per-tenant/control/KPI crossing state so a later breach is emitted again.
• Platform owns Notify conversion through AddNis2EffectivenessNotifyThresholdEmitter, which enqueues nis2.effectiveness.threshold_breached using the Notify queue contract. Telemetry hosts without Notify wiring keep the no-op emitter so local/offline metric collection remains deterministic.
• The v1 breach DTO carries tenant, area, control id, KPI key, metric name, observed value, target, severity, evidence mode/ref, default target hash, and responsible-role defaults. Target override and control-register snapshot hashes remain nullable until the downstream N1 sources are live.
• Offline Kit packages the Grafana dashboard JSON at telemetry/dashboards/nis2-effectiveness-dashboard.json from docs/modules/telemetry/operations/dashboards/nis2-effectiveness-dashboard.json; the OUK manifest inventories the file digest automatically.
• Federation remains opt-in through the existing FederatedTelemetry consent lifecycle. NIS2 KPI measurements are local Prometheus/OTLP metrics unless an administrator grants federation consent and sealed mode is disabled; there is no separate NIS2-specific federation switch in v1.

Contract:

• docs/contracts/nis2-kpi-telemetry-schema-v1.md

Verification coverage:

• src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core.Tests/Nis2AreaMetricsServiceTests.cs
• src/Platform/__Tests/StellaOps.Platform.WebService.Tests/Nis2TelemetryEndpointsTests.cs

Direct Policy, Findings, Authority, and Crypto call-site adoption is downstream module work; Telemetry Core owns the metric catalog, bounded emission, and deterministic reporting surface.

11) Federation DSSE Security Posture (Updated 2026-04-28)

Status:

• Advisory gap TEL-001 is closed. Federation consent and bundle paths now emit signed DSSE envelopes instead of payload passthrough placeholders.

Implemented contract:

• Consent and bundle envelopes now use explicit DSSE JSON structure: payloadType, base64 payload, and signatures[] (keyid, sig).
• Consent proofs and bundle summaries carry signer identity metadata (SignerKeyId) for auditability.
• Bundle payload canonicalization is deterministic for identical logical inputs:
  • bucket ordering: cveId (ordinal), then noisyCount (descending), artifactCount, observationCount
  • deterministic bundle ID derivation from canonical payload seed + fixed clock input
• Bundle verification enforces:
  • envelope digest integrity (sha256: over envelope bytes)
  • payload type match
  • trusted-key signature verification
  • consent digest linkage (consentDigest in payload must match ConsentDsseDigest)

Signer/verifier integration and fallback:

• Federation now uses explicit abstractions:
  • IFederationDsseEnvelopeSigner
  • IFederationDsseEnvelopeVerifier
• Default adapter: HmacFederationDsseEnvelopeService (offline-safe HMAC-SHA256 DSSE sign/verify using the local trusted key map in FederatedTelemetryOptions).
• No default signer secret or trusted key is shipped. Operators must configure DsseSignerKeyId, DsseSignerSecret, and trusted keys explicitly, or signing fails closed with federation.dsse.signer_unavailable.
• Failure mode is deterministic and auditable:
  • signing failures throw FederationSignatureException with stable error codes (for example federation.dsse.sign_failed, federation.dsse.signer_unavailable)
  • optional unsigned fallback (AllowUnsignedDsseFallback) emits envelopes tagged with offline-unsigned-fallback for explicit operator visibility.
• Local sync completion requires an outbox enqueue acknowledgement. FederatedTelemetrySyncService aggregates, signs, checks egress, then calls IFederatedTelemetryOutbox.EnqueueAsync; Synced means the durable local outbox accepted the bundle. It does not mean a peer acknowledged delivery. The default outbox remains unavailable and enabled startup rejects it.

Verification evidence:

• dotnet test src/Telemetry/StellaOps.Telemetry.Federation.Tests/StellaOps.Telemetry.Federation.Tests.csproj -m:1 -v minimal
• Coverage includes payload tamper, signature tamper, wrong-key verification failure, consent expiry + signature validity combination, and deterministic replay digest checks.

Tracking sprint:

• docs/implplan/SPRINT_20260304_307_Telemetry_federation_dsse_bundle_hardening.md

12) Decisions & Risks

• DORA Elite lead-time threshold — reconciled 2026-07-12 (CLO-7). DoraMetrics.ClassifyPerformance (src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core/DoraMetrics.cs) gates Elite on leadTimeHours < 24 (matching DoraMetricsOptions.LeadTimeSloHours = 24.0). The inline comment on that branch and the DoraPerformanceLevel.Elite XML doc in DoraMetricsModels.cs previously described Elite as “<1 hour lead time”; both were corrected to <24h so the comments no longer contradict the behaviour. Comment-only change; no runtime impact.