Federated Telemetry Operations Runbook

Audience: Platform operators configuring or troubleshooting the Platform federated-telemetry API.

Current maturity and ownership

Federated telemetry is a layered capability with local durable processing and an optional authenticated peer transport:

Synchronization is disabled by default. When enabled, Platform requires its durable PostgreSQL consent, fact, consent-set, privacy, bundle, outbox, and intelligence stores; startup fails with federation.startup.durable_prerequisites_unavailable if a process-local fallback remains selected. The worker does not use a synthetic default tenant or create consent.

Current component status:

Pending outbox rows are not peer acknowledgements. Do not claim live peer synchronization until a deployed two-installation mTLS forcing run produces the deterministic peer receipt.

Configuration boundary

Telemetry federation binds exclusively from Platform:TelemetryFederation:

{
  "Platform": {
    "TelemetryFederation": {
      "Enabled": false,
      "SealedModeEnabled": false,
      "SiteId": "site-001",
      "DestinationSiteId": "site-002",
      "EgressAllowedDestinationSiteIds": ["site-002"],
      "KAnonymityThreshold": 5,
      "EpsilonBudget": 1.0,
      "BudgetResetPeriod": "24:00:00",
      "AggregationInterval": "00:15:00",
      "DsseSignerKeyId": "secret-ref-managed-key-id",
      "DsseSignerSecret": "",
      "DsseTrustedKeys": {
        "site-002-key": "<secret-from-protected-provider>"
      },
      "PeerSignerKeySiteBindings": {
        "site-002-key": "site-002"
      },
      "PeerTransportEnabled": false,
      "PeerEndpoint": "https://peer-platform.internal:18444",
      "PeerClientCertificatePath": "/run/stellaops/tls/site-001-peer.p12",
      "PeerCertificateSiteBindings": {
        "<SHA256_CERTIFICATE_THUMBPRINT>": "site-002"
      }
    }
  }
}

STELLAOPS_FEATURE_FEDERATION, STELLAOPS_FEDERATION_REGION, and ReleaseOrchestrator:Federation:* configure the separate regional-federation controller in the ReleaseOrchestrator host. They do not belong to Platform and do not enable or configure telemetry federation. Gateway routes /api/v1/federation/* to ReleaseOrchestrator; Platform’s telemetry API remains under /api/v1/telemetry/federation/* and binds only Platform:TelemetryFederation.

The bundled RO regional-federation host is read-only. Its public mutation routes return HTTP 503 because mutation execution is not installed or implemented; no Platform telemetry setting can change that posture, and there is currently no operator-selectable RO execution provider.

The Platform host registers the background timer only when Platform:TelemetryFederation:Enabled=true. The HTTP endpoints remain mapped so operators can inspect consent and the fail-closed status.

Peer transport is separately disabled by default. When enabled, PeerEndpoint must be absolute HTTPS, the PKCS#12 certificate must be mounted, and the destination must match DestinationSiteId. The receiving Platform listener must request/require client certificates so Connection.ClientCertificate is populated; ordinary Gateway headers are ignored. Both Kestrel and the outbound .NET HTTP client must trust the installation’s private CA bundle; certificate fingerprint binding supplements CA validation and does not replace it. Startup also requires non-empty certificate-to-site, trusted-key, and signer-key-to-site maps. A peer DSSE key is accepted only when PeerSignerKeySiteBindings binds that key to the same source site authenticated by the client certificate. Supply certificate and DSSE secrets through protected configuration, not committed JSON.

For environment configuration, express dictionary entries as indexed Key/Value pairs (for example, Platform__TelemetryFederation__DsseTrustedKeys__0__Key=site-002-key and ...__0__Value=<secret>). Do not interpolate operator IDs into environment variable names: POSIX shell entrypoints can discard names containing hyphens before Platform starts. JSON and YAML providers may continue to use the direct map form shown above. Duplicate or blank indexed keys and values fail startup.

Compose deployment

Use the opt-in overlay after the base stack file:

docker compose -p stellaops `
  -f devops/compose/docker-compose.stella-ops.yml `
  -f devops/compose/docker-compose.telemetry-federation-peer.override.yml `
  -f devops/compose/docker-compose.existing-networks.override.yml `
  config --quiet

docker compose -p stellaops `
  -f devops/compose/docker-compose.stella-ops.yml `
  -f devops/compose/docker-compose.telemetry-federation-peer.override.yml `
  -f devops/compose/docker-compose.existing-networks.override.yml `
  up -d --force-recreate --no-deps platform

The overlay keeps the ordinary internal HTTP listener on 8080 and adds a direct Platform HTTPS listener on container port 8444, published as STELLAOPS_TELEMETRY_FEDERATION_LISTEN_PORT (default 18444). It sets ClientCertificateMode=RequireCertificate and SSL_CERT_FILE to the private CA bundle in STELLAOPS_TELEMETRY_FEDERATION_CERT_DIR. Configure the peer endpoint to this direct listener, never to the Gateway. Restrict the published port to peer installation addresses at the host firewall; the sample bind address is loopback-only and must be changed deliberately for cross-host installations.

The private certificate directory must contain:

Set the accepted peer client certificate’s uppercase, separator-free SHA-256 fingerprint in STELLAOPS_TELEMETRY_FEDERATION_SOURCE_CERT_SHA256 and bind it to the exact signed source site. Set that peer’s key ID and protected secret in STELLAOPS_TELEMETRY_FEDERATION_TRUSTED_DSSE_KEY_ID and STELLAOPS_TELEMETRY_FEDERATION_TRUSTED_DSSE_SECRET; the overlay binds the same key ID to STELLAOPS_TELEMETRY_FEDERATION_SOURCE_SITE_ID. A missing certificate must fail the TLS handshake; an untrusted certificate must fail chain validation; a CA-trusted but unbound certificate reaches the application and receives HTTP 401. A certificate-bound site presenting a bundle signed by a key bound to another site receives HTTP 422.

Authorization and scope

Consent and fact participation are tenant-owned. Site status, privacy budget, bundles, intelligence, and trigger are installation operations. Grant platform:federation:* only to installation operators.

The Console requests telemetry:participation:read|write and platform:federation:read|write, and the stella-ops-ui client allows them. No professional role bundle carries them (Executive Assurance Viewer, Release Operations Steward, Security Risk Manager, Service Release Contributor), so a Console session receives them through an administrator-family role, or through any role that Authority issuance does not recognize: such a session, including one with only the seeded operator or auditor role, receives every requested scope the client allows. Do not use an unrecognized custom role to model a restricted federation reader.

Read the current tenant state:

GET /api/v1/telemetry/federation/participation/consent

Grant or revoke current-tenant consent:

POST /api/v1/telemetry/federation/participation/consent/grant
Content-Type: application/json

{ "ttlHours": 720 }

POST /api/v1/telemetry/federation/participation/consent/revoke
Content-Type: application/json

{}

Consent signing fails closed with federation.dsse.signer_unavailable when no signer material is configured. Grant and revoke actors are derived from authenticated identity and proofs/revocations persist in Platform PostgreSQL. Consent alone does not authorize synchronization.

Authenticated runtime producers submit a current-tenant fact with telemetry:facts:write:

POST /api/v1/telemetry/federation/participation/facts
Content-Type: application/json

{ "artifactDigest": "sha256:<64 hex>", "cveId": "CVE-2026-7777", "symbolPath": "lib/example.so", "exploited": true, "observedAt": "2026-07-11T08:00:00Z" }

Tenant and accepting actor are derived from authenticated context; request JSON cannot select either.

Status and privacy budget

GET /api/v1/telemetry/federation/installation/status
GET /api/v1/telemetry/federation/installation/privacy-budget

status.enabled reflects Platform:TelemetryFederation:Enabled; sealedMode is reported separately. Installation status contains no tenant-consent field; inspect participation consent through the tenant route.

The privacy ledger is durable and uses an atomic conditional PostgreSQL update so concurrent workers cannot overspend the installation budget.

Trigger behavior

POST /api/v1/telemetry/federation/installation/trigger

Expected results:

ConfigurationResult
Enabled=false{ "triggered": false, "reason": "federation.sync.disabled" }
Enabled=true, SealedModeEnabled=true{ "triggered": false, "reason": "federation.sync.sealed_mode" }
Enabled=true, missing durable/configured prerequisite{ "triggered": false, "reason": "federation.sync.durable_participation_unavailable" } or startup validation failure
Enabled=true, configured, no eligible facts{ "triggered": false, "reason": "federation.sync.no_eligible_facts" }
Enabled=true, configured, eligible consenting tenant set{ "triggered": true }; bundle and pending outbox row are durable

triggered=true means accepted into the durable local outbox, not delivered to a peer. When peer transport is enabled, the leased delivery worker changes that row to delivered only after the configured destination returns the exact ${sourceSiteId}:${bundleId} receipt. Sealed mode keeps both aggregation and peer delivery network-silent.

Bundles and intelligence

GET /api/v1/telemetry/federation/installation/bundles
GET /api/v1/telemetry/federation/installation/bundles/{id}
GET /api/v1/telemetry/federation/installation/intelligence

Console pages

The Console reads the routes above and keeps these states separate. It never turns a failed read into a value.

StateHow the Console decides itWhat the page shows
Enabledstatus.enabled=trueInstallation Federation shows Enabled, the epsilon remaining this period and the recorded bundle count. Privacy Budget reads the ledger and shows it.
Disabledstatus.enabled=falseDisabled, with the statement that nothing is aggregated or synchronized and no privacy budget is spent. No epsilon figure is shown: for a period without spending, Platform reports the configured budget, which is not a measurement. Privacy Budget does not read the ledger. Enablement guidance (Platform:TelemetryFederation) is shown only to callers with platform:federation:write.
EmptyA successful read with no rowsBundles: “No federation bundles are recorded on this installation.” Intelligence: “No shared intelligence has been imported on this installation.” Both link to Installation Federation.
DeniedHTTP 401 or 403“… is not readable by this session” with the HTTP status and the owner error code when the response carries one (the gateway answers a missing scope with HTTP 403 and forbidden). HTTP 401 is described as an unauthenticated or expired session, HTTP 403 as a missing scope. No value, no action, no Retry.
UnavailableAny other HTTP status, no response, no answer within 10 seconds, or a response without the fields the page renders“… unavailable” with the cause and Retry. No value and no action.

Current Tenant Participation shows Consent Granted or Consent Not Granted only from a successful consent read. Grant and Revoke are offered only to callers with telemetry:participation:write; a failed grant or revoke names the HTTP status, the owner error code when the response carries one, and the problem detail of an HTTP 4xx answer, and leaves the displayed state as last read. The grant route does not catch DSSE signing failures, so a grant without signer key material is answered with HTTP 400 and the detail “Federation DSSE signer key material is not configured.” and carries no owner error code. Consent is tenant-scoped: a tenant switch clears the displayed consent and any action message, cancels a consent read in flight and reads the newly selected tenant’s consent; the answer to an action sent for the previous tenant is not shown, and an action is not sent until the new tenant’s consent has been read. The page does not read installation status; it states that consent records the tenant’s permission only and that aggregation and synchronization are an installation setting, and links Installation Federation only for callers with platform:federation:read|write.

The bundle list in the Console has no verification column, because the list route does not verify envelopes.

Troubleshooting

Never work around these states by granting consent to a default tenant, enabling unsigned DSSE fallback in production, or adding an in-memory fact producer.