Federated Telemetry Operations Runbook

Audience: Platform operators configuring or troubleshooting the Platform federated-telemetry API.

Current maturity and ownership

Federated telemetry is a layered capability with local durable processing and an optional authenticated peer transport:

Synchronization is disabled by default. When enabled, Platform requires its durable PostgreSQL consent, fact, consent-set, privacy, bundle, outbox, and intelligence stores; startup fails with federation.startup.durable_prerequisites_unavailable if a process-local fallback remains selected. The worker does not use a synthetic default tenant or create consent.

Current component status:

Pending outbox rows are not peer acknowledgements. Do not claim live peer synchronization until a deployed two-installation mTLS forcing run produces the deterministic peer receipt.

Configuration boundary

Telemetry federation binds exclusively from Platform:TelemetryFederation:

{
  "Platform": {
    "TelemetryFederation": {
      "Enabled": false,
      "SealedModeEnabled": false,
      "SiteId": "site-001",
      "DestinationSiteId": "site-002",
      "EgressAllowedDestinationSiteIds": ["site-002"],
      "KAnonymityThreshold": 5,
      "EpsilonBudget": 1.0,
      "BudgetResetPeriod": "24:00:00",
      "AggregationInterval": "00:15:00",
      "DsseSignerKeyId": "secret-ref-managed-key-id",
      "DsseSignerSecret": "",
      "DsseTrustedKeys": {
        "site-002-key": "<secret-from-protected-provider>"
      },
      "PeerSignerKeySiteBindings": {
        "site-002-key": "site-002"
      },
      "PeerTransportEnabled": false,
      "PeerEndpoint": "https://peer-platform.internal:18444",
      "PeerClientCertificatePath": "/run/stellaops/tls/site-001-peer.p12",
      "PeerCertificateSiteBindings": {
        "<SHA256_CERTIFICATE_THUMBPRINT>": "site-002"
      }
    }
  }
}

Platform:Federation and STELLAOPS_FEATURE_FEDERATION configure the separate ReleaseOrchestrator regional-federation controller. They do not enable or configure Telemetry federation.

The Platform host registers the background timer only when Platform:TelemetryFederation:Enabled=true. The HTTP endpoints remain mapped so operators can inspect consent and the fail-closed status.

Peer transport is separately disabled by default. When enabled, PeerEndpoint must be absolute HTTPS, the PKCS#12 certificate must be mounted, and the destination must match DestinationSiteId. The receiving Platform listener must request/require client certificates so Connection.ClientCertificate is populated; ordinary Gateway headers are ignored. Both Kestrel and the outbound .NET HTTP client must trust the installation’s private CA bundle; certificate fingerprint binding supplements CA validation and does not replace it. Startup also requires non-empty certificate-to-site, trusted-key, and signer-key-to-site maps. A peer DSSE key is accepted only when PeerSignerKeySiteBindings binds that key to the same source site authenticated by the client certificate. Supply certificate and DSSE secrets through protected configuration, not committed JSON.

For environment configuration, express dictionary entries as indexed Key/Value pairs (for example, Platform__TelemetryFederation__DsseTrustedKeys__0__Key=site-002-key and ...__0__Value=<secret>). Do not interpolate operator IDs into environment variable names: POSIX shell entrypoints can discard names containing hyphens before Platform starts. JSON and YAML providers may continue to use the direct map form shown above. Duplicate or blank indexed keys and values fail startup.

Compose deployment

Use the opt-in overlay after the base stack file:

docker compose -p compose `
  -f devops/compose/docker-compose.stella-ops.yml `
  -f devops/compose/docker-compose.telemetry-federation-peer.override.yml `
  -f devops/compose/docker-compose.existing-networks.override.yml `
  config --quiet

docker compose -p compose `
  -f devops/compose/docker-compose.stella-ops.yml `
  -f devops/compose/docker-compose.telemetry-federation-peer.override.yml `
  -f devops/compose/docker-compose.existing-networks.override.yml `
  up -d --force-recreate --no-deps platform

The overlay keeps the ordinary internal HTTP listener on 8080 and adds a direct Platform HTTPS listener on container port 8444, published as STELLAOPS_TELEMETRY_FEDERATION_LISTEN_PORT (default 18444). It sets ClientCertificateMode=RequireCertificate and SSL_CERT_FILE to the private CA bundle in STELLAOPS_TELEMETRY_FEDERATION_CERT_DIR. Configure the peer endpoint to this direct listener, never to the Gateway. Restrict the published port to peer installation addresses at the host firewall; the sample bind address is loopback-only and must be changed deliberately for cross-host installations.

The private certificate directory must contain:

Set the accepted peer client certificate’s uppercase, separator-free SHA-256 fingerprint in STELLAOPS_TELEMETRY_FEDERATION_SOURCE_CERT_SHA256 and bind it to the exact signed source site. Set that peer’s key ID and protected secret in STELLAOPS_TELEMETRY_FEDERATION_TRUSTED_DSSE_KEY_ID and STELLAOPS_TELEMETRY_FEDERATION_TRUSTED_DSSE_SECRET; the overlay binds the same key ID to STELLAOPS_TELEMETRY_FEDERATION_SOURCE_SITE_ID. A missing certificate must fail the TLS handshake; an untrusted certificate must fail chain validation; a CA-trusted but unbound certificate reaches the application and receives HTTP 401. A certificate-bound site presenting a bundle signed by a key bound to another site receives HTTP 422.

Authorization and scope

Consent and fact participation are tenant-owned. Site status, privacy budget, bundles, intelligence, and trigger are installation operations. Grant platform:federation:* only to installation operators.

Read the current tenant state:

GET /api/v1/telemetry/federation/participation/consent

Grant or revoke current-tenant consent:

POST /api/v1/telemetry/federation/participation/consent/grant
Content-Type: application/json

{ "ttlHours": 720 }

POST /api/v1/telemetry/federation/participation/consent/revoke
Content-Type: application/json

{}

Consent signing fails closed with federation.dsse.signer_unavailable when no signer material is configured. Grant and revoke actors are derived from authenticated identity and proofs/revocations persist in Platform PostgreSQL. Consent alone does not authorize synchronization.

Authenticated runtime producers submit a current-tenant fact with telemetry:facts:write:

POST /api/v1/telemetry/federation/participation/facts
Content-Type: application/json

{ "artifactDigest": "sha256:<64 hex>", "cveId": "CVE-2026-7777", "symbolPath": "lib/example.so", "exploited": true, "observedAt": "2026-07-11T08:00:00Z" }

Tenant and accepting actor are derived from authenticated context; request JSON cannot select either.

Status and privacy budget

GET /api/v1/telemetry/federation/installation/status
GET /api/v1/telemetry/federation/installation/privacy-budget

status.enabled reflects Platform:TelemetryFederation:Enabled; sealedMode is reported separately. Installation status contains no tenant-consent field; inspect participation consent through the tenant route.

The privacy ledger is durable and uses an atomic conditional PostgreSQL update so concurrent workers cannot overspend the installation budget.

Trigger behavior

POST /api/v1/telemetry/federation/installation/trigger

Expected results:

ConfigurationResult
Enabled=false{ "triggered": false, "reason": "federation.sync.disabled" }
Enabled=true, SealedModeEnabled=true{ "triggered": false, "reason": "federation.sync.sealed_mode" }
Enabled=true, missing durable/configured prerequisite{ "triggered": false, "reason": "federation.sync.durable_participation_unavailable" } or startup validation failure
Enabled=true, configured, no eligible facts{ "triggered": false, "reason": "federation.sync.no_eligible_facts" }
Enabled=true, configured, eligible consenting tenant set{ "triggered": true }; bundle and pending outbox row are durable

triggered=true means accepted into the durable local outbox, not delivered to a peer. When peer transport is enabled, the leased delivery worker changes that row to delivered only after the configured destination returns the exact ${sourceSiteId}:${bundleId} receipt. Sealed mode keeps both aggregation and peer delivery network-silent.

Bundles and intelligence

GET /api/v1/telemetry/federation/installation/bundles
GET /api/v1/telemetry/federation/installation/bundles/{id}
GET /api/v1/telemetry/federation/installation/intelligence

Troubleshooting

Never work around these states by granting consent to a default tenant, enabling unsigned DSSE fallback in production, or adding an in-memory fact producer.