RuntimeInstrumentation Host Agent Strategy

Decision Summary

RuntimeInstrumentation should be implemented as Stella Ops host agents for non-Kubernetes container estates. The product surface is:

• A Stella Ops Linux host agent.
• A Stella Ops Windows host agent.
• Shared Agent.Core contracts used by both agents.
• Optional Linux sensor inputs, including Tetragon where available.
• Digest-gated evidence publication into the Stella Ops control plane.

Tetragon is a Linux eBPF sensor and bridge. It is not the Stella Ops Linux agent. It can supply high-value Linux runtime observations to the Linux agent, but it does not own agent identity, install lifecycle, runtime status authority, evidence buffering, signing, policy configuration, or publication.

Kubernetes is not active scope. RuntimeInstrumentation must not add Helm charts, DaemonSets, Kubernetes watches, Kubernetes informers, or Kubernetes-first deployment instructions.

Product Split

Why Tetragon Is Not The Linux Agent

Tetragon observes Linux kernel/user-space activity through eBPF. That makes it useful for process execution, syscall, and stack/symbol observations on Linux hosts.

The Stella Ops Linux agent has a broader product job:

• Install and run as the Stella Ops controlled host component.
• Own Stella Ops configuration and policy-loaded settings.
• Identify the host and agent instance.
• Read local runtime status and image identity.
• Decide whether an observation is publishable.
• Buffer and retry evidence publication.
• Report health and diagnostics.
• Enforce digest-gated evidence rules.

Tetragon can feed observations into that agent. It does not replace that agent.

Implementation Path

Phase 1 - Shared Agent.Core Contracts

Create shared contracts before implementing OS-specific agents. This phase now has an initial implementation in src/RuntimeInstrumentation/StellaOps.Agent.Core/:

• AgentIdentity: agent id, host id, install id, version, OS, architecture, environment, and tenant/site scope.
• AgentCapability: declared capabilities such as runtime status, process observation, optional sensor bridge, publisher, local diagnostics, and offline spool.
• HostRuntimeStatus: source id, runtime kind, container id, image reference, image id, resolved digest, labels, and freshness metadata.
• RuntimeObservation: normalized observation from any local sensor, with source identity and host/container correlation fields.
• RuntimeEvidenceEnvelope: publishable evidence with digest, deterministic event id, sequence, source identity, agent identity, timestamp, and verification metadata.
• IRuntimeStatusSource: refresh/read local runtime status without knowing about downstream publication.
• IRuntimeObservationSource: read local observations from approved sensors.
• IRuntimeEvidencePublisher: publish, retry, quarantine, and expose health.
• AgentCoreValidation, RuntimeDigest, RuntimeEvidenceIds, and AgentCoreJson: validation, digest gate, deterministic ids, and deterministic serialization helpers.

Acceptance expectations:

• Contracts contain no Kubernetes types.
• Contracts contain no Tetragon-specific types.
• Serialization is deterministic and covered by tests.
• Validation rejects missing host id, missing source id, missing digest on publishable evidence, malformed digest, and unsupported OS/runtime values.

Phase 2 - Linux Host Agent

Implement the Linux agent as a Stella Ops owned host service. The base service now lives in src/RuntimeInstrumentation/StellaOps.Agent.Linux/.

Core responsibilities:

• Service host and packaging direction for systemd-managed deployment.
• Configuration loading from files, environment, and policy-delivered settings with explicit precedence.
• Local runtime status collection from approved file-backed and command-backed sources first.
• Future native runtime APIs only after CRI/runtime contracts and license review are complete.
• Optional sensor loading, including the Tetragon bridge when explicitly enabled.
• Digest gate before publication.
• Local bounded evidence spool.
• Health, metrics, and diagnostics.

Acceptance expectations:

• The agent starts without optional observation sources.
• The agent fails closed when runtime status is missing, stale, malformed, tag-only, ambiguous, permission-denied, timed out, or over-large.
• The base health/status snapshot contract and bounded local spool never accept publishable status without a valid sha256:<64-hex> digest.
• Local commands are fully qualified, never shell-invoked, timeout bounded, and output bounded.
• Unit tests cover configuration, status parsing, digest gating, spool behavior, retry behavior, unsupported platform behavior, health/status snapshots, absent optional sensors, and digest-gated Agent.Core evidence envelopes.

The base now publishes digest-valid local snapshots through the Agent.Core IRuntimeEvidencePublisher boundary. If no downstream publisher is configured, the default no-op publisher returns a retry result and the entry remains in the bounded spool.

Phase 3 - Windows Host Agent

Implement the Windows agent as a Stella Ops owned Windows Service. The base now lives in src/RuntimeInstrumentation/StellaOps.Agent.Windows/.

Core responsibilities:

• Windows Service lifecycle and install/uninstall behavior.
• Configuration loading from files, environment, and policy-delivered settings with explicit precedence.
• Local Windows runtime status from approved Windows runtime APIs, named pipes, file-backed snapshots, or bounded commands.
• Windows process/container correlation.
• Digest gate before publication.
• Local bounded evidence spool.
• Windows-native health, event logging, diagnostics, ACLs, and credential handling.

Acceptance expectations:

• The agent does not depend on Tetragon.
• The agent uses Windows-native service and permission boundaries.
• The same digest-gated publication contract is used as Linux.
• Unit tests cover configuration, status parsing, digest gating, spool behavior, retry behavior, and Windows permission failures.

Current base behavior:

• Generic-host service entry point with Windows Service lifetime integration and service name Stella Ops RuntimeInstrumentation Agent, with a Windows-only platform guard.
• File -> environment -> policy configuration precedence using STELLAOPS_WINDOWS_AGENT_ environment variables.
• File-backed and bounded command-backed Windows runtime status sources implementing Agent.Core IRuntimeStatusSource.
• Command execution uses fully qualified executable paths, typed arguments, UseShellExecute=false, explicit timeouts, and output byte limits.
• Digest gate blocks missing, malformed, tag-only, stale, and ambiguous status before the bounded Windows spool.
• The base publishes local health/status snapshots and digest-valid Agent.Core evidence envelopes. If no downstream publisher is configured, the default no-op publisher returns a retry result and the entry remains in the bounded spool.

Phase 4 - Optional Tetragon Bridge

Implement Tetragon as a Linux-only optional observation source.

Core responsibilities:

• Read local Tetragon health, policy state, and event output through an approved bounded transport.
• Normalize Tetragon events into Agent.Core RuntimeObservation records.
• Report dropped events, lag, health, and source trust to the Linux agent.
• Hand observations to the Linux agent; do not publish directly.
• Keep Tetragon disabled by default unless explicitly configured.

Acceptance expectations:

• Tetragon bridge is not required for Linux agent startup.
• Tetragon bridge does not introduce Kubernetes deployment assumptions.
• Backpressure, maximum event size, malformed events, disconnects, and disabled sensor behavior are test-covered.
• Tetragon-specific fields do not leak into Agent.Core contracts.

Current base behavior:

• src/RuntimeInstrumentation/StellaOps.Agent.Tetragon/ implements TetragonRuntimeObservationSource as an Agent.Core IRuntimeObservationSource.
• The bridge is disabled unless RuntimeInstrumentation:LinuxAgent:TetragonBridge:Enabled=true; disabled or absent Tetragon does not register an observation source and does not block Linux agent startup.
• The Linux agent reads optional observation sources into its local snapshot surface with a bounded MaxObservationsPerCycle limit, stamps observations with Linux agent/host identity, and owns downstream evidence publication through Agent.Core. The Tetragon bridge never publishes evidence directly.
• The bridge maps supported local Tetragon event output to Agent.Core RuntimeObservation records and exposes Tetragon details only as attributes.
• Unit tests cover disabled behavior, process-event adaptation, pull-based backpressure, malformed events, oversized event drops, stream disconnects, bridge health counters, and Linux agent operation with Tetragon absent.

Phase 5 - CRI And Local Runtime Status Sources

Finish runtime status authority in layers:

1. Keep file-backed and command-backed local status sources as the deterministic baseline.
2. Add native Linux CRI/container runtime clients only after protobuf/gRPC or runtime API contracts are owned and license-approved.
3. Add native Windows runtime clients only after Windows API ownership, permissions, and fixtures are approved.
4. Keep each source optional and independently testable.

Acceptance expectations:

• Container ids remain lookup keys only.
• Tags never become image evidence.
• Runtime status freshness is included in publication decisions.
• Native clients have fixtures for success, not-found, permission denied, timeout, malformed response, tag-only response, and stale response.
• Source authority, permissions, freshness, and failure behavior are tracked in Runtime Status Source Matrix.

Phase 6 - Digest-Gated Evidence Publication

Implement evidence publication after local status and observation contracts are stable.

Core responsibilities:

• Build deterministic evidence envelopes.
• Sign or attest envelopes according to the platform evidence mode.
• Retry transient failures.
• Quarantine invalid/unpublishable observations with clear reason codes.
• Keep bounded local retention.
• Emit health and metrics for publish, retry, quarantine, and drop paths.

Acceptance expectations:

• No evidence leaves the host without a valid digest.
• Duplicate observations are idempotent.
• Replay metadata is sufficient for audit.
• Retention overflow behavior is explicit and tested.
• Downstream services can verify source identity, digest identity, and event ordering.

Current implementation:

• Linux and Windows agents construct RuntimeEvidenceEnvelope records from digest-valid local snapshot entries and publish them through IRuntimeEvidencePublisher.
• Envelopes carry digest, source identity, host identity, agent identity, event id, sequence, capture timestamp, runtime status freshness, verification metadata, and replay metadata.
• Publish, retry, quarantine, drop, duplicate, and retention overflow paths are surfaced in local snapshot diagnostics.
• AgentCoreValidation rejects malformed envelopes and explicitly unsigned evidence. No downstream service verification contract exists in RuntimeInstrumentation scope yet, so service-specific cryptographic verification remains a future Evidence/Attestor integration.

Phase 7 - Tests And CI

Add verification in layers:

• Contract tests for Agent.Core serialization and validation.
• Unit tests for Linux and Windows configuration, runtime status parsing, digest gating, publisher behavior, and spool behavior.
• Fixture tests for local status documents and command outputs.
• Linux integration tests on a pinned Linux runner for service startup, permissions, local command execution, and optional Tetragon bridge behavior.
• Windows integration tests on a pinned Windows runner for service startup, ACLs, named pipe/runtime source behavior, and publisher retry behavior.
• Negative tests proving Kubernetes is not required and Helm artifacts are not part of RuntimeInstrumentation release validation.

CI expectations:

• RuntimeInstrumentation CI must not require Kubernetes.
• RuntimeInstrumentation CI must not require Helm.
• Tetragon integration tests are optional Linux-runner tests and must be skippable when the runner lacks eBPF permissions, while unit/fixture tests remain mandatory.

Current CI implementation:

• .gitea/workflows/runtime-instrumentation-host-agents.yml runs regular Agent.Core, Linux host-agent, Windows host-agent, and optional Tetragon bridge unit/fixture tests.
• Optional self-hosted Linux and Windows integration lanes are manual-dispatch only and record runner facts before running host-agent tests.
• The workflow has no Kubernetes or Helm prerequisite and does not package RuntimeInstrumentation as charts, DaemonSets, or Kubernetes manifests.

Implementation Risks

• Native CRI support can become fake authority if implemented without owned protobuf/generated contracts or approved runtime APIs.
• Tetragon can accidentally become the perceived Linux agent if docs, package names, and capability registration are not explicit.
• Windows parity can lag if Agent.Core contracts assume Linux process/container semantics.
• Evidence publication can create audit gaps if invalid observations are silently dropped instead of reason-coded and counted.

Product Decisions Recorded

• Stella Ops does not support Kubernetes or Helm as active RuntimeInstrumentation deployment scope.
• Stella Ops Linux and Windows agents are the product capabilities.
• Tetragon is optional Linux sensor input only.
• Runtime evidence must be digest-gated.
• IRandomProvider work is separate determinism infrastructure and should not be mixed into RuntimeInstrumentation host-agent contracts unless a concrete runtime-agent need appears.