Runtime Instrumentation Architecture

Active architecture for the Stella Ops Linux and Windows host agents — local runtime status collection, optional Linux Tetragon sensor input, and digest-gated evidence publication. Audience: engineers building the host agents and the shared Agent.Core contracts, and operators deploying agents onto non-Kubernetes container hosts. This is a host-agent architecture, not a Kubernetes or Tetragon-first one.

Overview

Runtime Instrumentation connects local host runtime observations to Stella Ops release and security decisions. The active architecture is not Tetragon-first and not Kubernetes-first. It is a host-agent architecture for non-Kubernetes container estates.

The Stella Ops Linux agent and Stella Ops Windows agent are the deployable products. They collect local container runtime status, normalize local sensor observations, enforce digest-gated evidence rules, and publish evidence to Stella Ops services. Tetragon is one optional Linux eBPF sensor input that the Linux agent may bridge when it is available; it is not the Stella Ops Linux agent.

Product Boundary

Active scope:

• Linux host agent for non-Kubernetes Linux container hosts.
• Windows host agent for non-Kubernetes Windows container hosts.
• Shared Agent.Core contracts used by both agents.
• Local runtime status sources: snapshot files, bounded local commands, and future approved native runtime APIs.
• Optional Linux Tetragon bridge as a sensor input to the Linux agent.
• Optional Linux file-backed stack-frame symbol resolution (co-registered with the Tetragon bridge).
• Digest-gated evidence publication to downstream Stella Ops services.

Out of active scope:

• Kubernetes deployment support.
• Helm charts.
• Kubernetes DaemonSets.
• Kubernetes API informers, watches, or pod-status polling as product direction.
• Treating Tetragon as the Stella Ops Linux agent.

Existing Kubernetes-related source slices are historical and must not be expanded as part of the host-agent plan.

Terminology

Target Components

RuntimeInstrumentation/
  StellaOps.Agent.Core/                         # Shared host-agent contracts
    AgentIdentity
    AgentCapability
    HostRuntimeStatus
    RuntimeObservation
    RuntimeStatusSource
    RuntimeEvidencePublisher
    AgentHealth

  StellaOps.Agent.Linux/                        # Linux host-agent base
    generic-host daemon entry point
    Linux runtime status sources
    optional local sensor coordination boundary
    bounded local status spool
    health/status snapshot publisher contract

  StellaOps.Agent.Windows/                      # Windows host-agent base
    Windows Service-style generic-host lifecycle
    Windows runtime status sources
    optional Windows sensor coordination boundary
    bounded local status spool
    health/status snapshot publisher contract

  StellaOps.Agent.Linux.SymbolResolver/         # Linux file-backed stack-frame symbolizer
    IProcessSymbolResolver / ProcessSymbolResolver
    RuntimeObservationSymbolUpgrader
    Internal/ProcMapsParser, ElfSymbolTableReader, DwarfLineReader
    AddStellaOpsLinuxSymbolResolver registration

  StellaOps.Agent.Tetragon/                     # Optional Linux Tetragon bridge/client
    TetragonExportClient (named HttpClient "StellaOps.Agent.Tetragon")
    TetragonRuntimeObservationSource (IRuntimeObservationSource + ITetragonBridgeHealth)
    TetragonRuntimeObservationAdapter (event normalization)
    bridge into Agent.Core observation contracts

  StellaOps.RuntimeInstrumentation.Tetragon/    # Existing adapter model slice
    ContainerImageDigestResolver
    RuntimeContainerImageStatus command/Kubernetes API/Kubernetes watch sources
    canonical runtime container-image status snapshots

  StellaOps.RuntimeInstrumentation.Worker/      # Existing local status worker slice
    RuntimeInstrumentationWorkerHost
    local file/command status refresh
    digest resolver wiring

The target components are incremental. Current code already contains the shared StellaOps.Agent.Core contract boundary, Linux and Windows host-agent bases (including the digest gate, bounded spool, and evidence-envelope publication path), the file-backed Linux symbol resolver, the optional Tetragon bridge/client, worker wiring, and the legacy StellaOps.RuntimeInstrumentation.Tetragon adapter slice (which still contains Kubernetes API/watch source classes that are historical and out of active scope). The remaining product boundary is the production downstream IRuntimeEvidencePublisher implementation (Signals/Scanner/Policy/EvidenceLocker/Attestor wiring); the default registration is NoopRuntimeEvidencePublisher.

Data Flow

[Linux container host]                         [Windows container host]
        |                                                |
        v                                                v
[Stella Ops Linux agent]                       [Stella Ops Windows agent]
        |                                                |
        | local runtime status                           | local runtime status
        | optional Tetragon sensor input                 | Windows runtime/sensor inputs
        v                                                v
[Agent.Core observation contracts]             [Agent.Core observation contracts]
        |                                                |
        +--------------------+---------------------------+
                             v
                    [Digest gate]
                             |
              only sha256 digest-backed evidence
                             v
                    [Evidence publisher]
                             |
        +--------------------+---------------------------+
        v                    v                           v
    [Signals]            [Scanner]                    [Policy]

Tetragon appears only inside the Linux host path as an optional sensor input:

[Tetragon on Linux host]
        |
        v
[Tetragon bridge]
        |
        v
[Stella Ops Linux agent]

If Tetragon is absent, unsupported, or disabled, the Linux agent must still operate with local runtime status and any other approved local sensor inputs.

Agent.Core Contracts

Agent.Core defines stable contracts before the Linux and Windows agents diverge. The source lives under src/RuntimeInstrumentation/StellaOps.Agent.Core/, with contract tests under src/RuntimeInstrumentation/StellaOps.Agent.Core.Tests/.

• Agent identity (AgentIdentity): stable agent id, host id, install id, version, AgentOperatingSystem (Linux = 1, Windows = 2), architecture, environment, optional tenant, and optional site scope. Host identity is a separate HostIdentity record (host id, hostname, OS, architecture, optional site, metadata).
• Capability declaration (AgentCapability keyed by AgentCapabilityKind): RuntimeStatus = 1, ProcessObservation = 2, SensorBridge = 3, EvidencePublication = 4, HealthReporting = 5, LocalDiagnostics = 6, OfflineSpool = 7.
• Runtime status model (HostRuntimeStatus): source id, RuntimeKind (Docker = 1, Containerd = 2, Cri = 3, Podman = 4, WindowsContainers = 5, FileSnapshot = 6, CommandSnapshot = 7), container id, image reference, image id, image digest when available, observed-at timestamp, freshness, labels, and metadata. HasPublishableDigest is true only when ImageDigest is a valid sha256:<64-hex> value.
• Runtime observation model (RuntimeObservation): observation id, RuntimeObservationKind (ProcessStarted = 1, ProcessExited = 2, NetworkConnection = 3, FileAccess = 4, RuntimeStatusChanged = 5, SensorHealthChanged = 6), source id, agent id, host id, container id, observed-at timestamp, optional process id/name, attributes, and optional stack/symbol data. Stack-frame payload (Sprint 022 PHASE-B): optional UserStack, KernelStack lists of RuntimeObservationStackFrame { ulong InstructionAddress, ulong? Offset, string? Module, string? Symbol, uint? Line }, plus a wire-stable StackCaptureMode string. The mode constants are defined in StackCaptureModes: none ("none"), addresses-only ("addresses-only"), partial-symbols ("partial-symbols"), full-symbols ("full-symbols"); it is a string rather than a CLR enum so newer producers can add modes without breaking older consumers. All stack fields nullable; default behaviour is byte-identical to pre-PHASE-B. Validation bounds (constants MaxStackFramesPerList, MaxSymbolLength, MaxModulePathLength, MaxStackPayloadBytes): 256 frames per list, 512-char symbols, 1024-char module paths, 16 KB total estimated stack payload. Deterministic event-id (RuntimeEvidenceIds.CreateEventId, prefix rtevt:) incorporates a SHA-256 digest of the canonical stack JSON only when stack fields are present, preserving back-compat for legacy callers.
• Evidence envelope (RuntimeEvidenceEnvelope): event id, sequence number, source id, agent identity, host identity, observation, runtime status, image digest, captured-at timestamp, runtime-status freshness, optional verification metadata, and optional replay metadata. The publish result (RuntimeEvidencePublishResult) carries a RuntimeEvidencePublishDisposition (Published = 1, Retried = 2, Quarantined = 3, Dropped = 4), a reason string, optional receipt id, and optional retry-after.
• Publisher contract (IRuntimeEvidencePublisher): PublishAsync and QuarantineAsync. Status sources implement IRuntimeStatusSource (pull) and optional sensor sources implement IRuntimeObservationSource (streaming IAsyncEnumerable). AgentHealth reports AgentHealthState (Healthy = 1, Degraded = 2, Unhealthy = 3, Starting = 4, Stopping = 5).
• Configuration contract (AgentConfiguration): separate file, environment, and policy settings dictionaries plus a config version, supporting explicit precedence.
• Deterministic support: AgentCoreJson.SerializeDeterministic (camelCase enums, null-omitting, recursively key-sorted JSON) and deterministic runtime evidence event-id creation.
• Validation support (AgentCoreValidation): contract validation rejects missing identity fields, unsupported OS/runtime/observation enum values, malformed image digests, negative freshness, non-positive evidence sequences, missing capture timestamps, digest mismatches between envelope and runtime status, observation/agent id mismatches, missing verification metadata, a digest gate that is not passed, a missing attestation mode, and explicitly unsigned attestation. Stack validation enforces the frame/symbol/module/payload bounds and the mode-vs-frames consistency rules (e.g. addresses-only forbids symbols; partial-symbols/full-symbols require at least one symbol). Note the digest format check (RuntimeDigest.IsSha256Digest) accepts upper- or lower-case hex.

Agent.Core must be free of Kubernetes types and free of Tetragon-specific types.

Linux Host Agent

The Linux host agent should be a Stella Ops owned service, normally installed through package/service tooling rather than Kubernetes resources.

Responsibilities:

• Run as a managed service with least privilege.
• Read local runtime status from approved sources such as containerd/CRI, Docker, Podman, snapshot files, or bounded commands.
• Map process/container observations to runtime status.
• Optionally load the Tetragon bridge when Tetragon is installed and enabled.
• Enforce digest-gated publication before runtime evidence leaves the host.
• Expose health, metrics, local diagnostics, and configuration validation.
• Buffer evidence locally when the central Stella Ops service is unavailable, within bounded retention limits.

The Linux agent is not Tetragon. Tetragon can improve Linux visibility, but the agent must remain useful without it.

Current base implementation:

• src/RuntimeInstrumentation/StellaOps.Agent.Linux/ provides the generic-host daemon entry point and DI wiring via AddStellaOpsLinuxHostAgent. Options bind from the RuntimeInstrumentation:LinuxAgent configuration section.
• Startup options validation (LinuxHostAgentOptionsValidator) requires agent id, host id, install id, version, environment, config version, positive collection interval and max-runtime-status-age, non-negative MaxObservationsPerCycle, positive spool capacity, non-negative spool retry attempts, and at least one local file or command runtime status source. File/command sources additionally require fully-qualified paths, defined DocumentKind/RuntimeKind enums, positive command timeouts, and per-command MaxOutputBytes within bounds (≤ 16 MiB).
• Configuration precedence is explicit: local files (/etc/stellaops/linux-agent.json then linux-agent.json) load first, STELLAOPS_LINUX_AGENT_ environment variables (with __ → : key mapping) override file values, and an optional policy-delivered JSON file (RuntimeInstrumentation:LinuxAgent:PolicyFile) is loaded last.
• File-backed and command-backed runtime status sources implement Agent.Core IRuntimeStatusSource; command execution uses ProcessStartInfo.ArgumentList, UseShellExecute=false, timeout bounds, and stdout/stderr byte limits.
• An optional observation-filter pipeline (IRuntimeObservationFilter, registered in DI order) transforms streamed RuntimeObservation records before they are enqueued for publication. Filters must never drop observations; if a filter throws, the agent logs an observation_filter.failed issue and falls back to the last successful version of the observation. The only built-in filter, SymbolResolutionObservationFilter, is co-registered with the Tetragon bridge (see Optional Tetragon Bridge) and is absent when the bridge is disabled.
• The Linux-only platform guard (LinuxHostAgentIdentityFactory.EnsureLinuxPlatform) throws PlatformNotSupportedException before startup on non-Linux platforms.
• The host identity carries fixed metadata agent.kind=linux-host and runtimeInstrumentation.scope=non-kubernetes.
• The base emits a local LinuxHostAgentSnapshot containing Agent.Core identity, host identity, configuration, capabilities, health, publishable runtime statuses, blocked runtime statuses, runtime observations, source issues, observation-source issues, and diagnostics. The default snapshot publisher is NoopLinuxHostAgentSnapshotPublisher.
• The digest gate (LinuxRuntimeStatusGate.Classify) blocks each status with a deterministic reason code: runtime_status.digest_missing_or_malformed, runtime_status.ambiguous (two records on the same normalized container id, stripping containerd:///docker:///cri-o:// prefixes), or runtime_status.stale (freshness exceeds MaxRuntimeStatusAge).
• The local spool accepts only HostRuntimeStatus records with valid sha256:<64-hex> digests and enforces capacity and retry-attempt bounds.
• Digest-valid spool entries are converted into Agent.Core RuntimeEvidenceEnvelope records (event id from RuntimeEvidenceIds.CreateEventId) and sent to the configured IRuntimeEvidencePublisher. Each envelope carries verification metadata digest.gate=passed, attestation.mode=local-digest-gated, evidence.publisher=linux-host-agent, and replay metadata agent.core.schema=runtime-evidence-envelope-v1. Envelopes failing AgentCoreValidation are quarantined rather than published.
• Publication diagnostics record attempted, published, retried, quarantined, dropped, duplicate, and last-reason counters in the local snapshot.

Windows Host Agent

The Windows host agent is a Stella Ops owned Windows Service-style host base, normally installed through Windows service/package tooling rather than Kubernetes resources.

Responsibilities:

• Read local Windows container runtime status from approved Windows runtime APIs, named pipes, command-backed snapshots, or file-backed snapshots.
• Map Windows process/container observations to runtime status.
• Enforce the same digest-gated publication rules as Linux.
• Use Windows-native service control, ACL, credential storage, logging, and eventing patterns.
• Expose health, diagnostics, and bounded local evidence buffering.

Tetragon is Linux-only and must not be used as a Windows agent dependency.

Current base implementation:

• src/RuntimeInstrumentation/StellaOps.Agent.Windows/ provides the generic-host service entry point, Windows Service lifetime integration, and DI wiring (AddStellaOpsWindowsHostAgent) for Windows Service packaging. Options bind from the RuntimeInstrumentation:WindowsAgent configuration section. The Windows Service name is Stella Ops RuntimeInstrumentation Agent (set in Program.cs). Host identity carries metadata agent.kind=windows-host, runtimeInstrumentation.scope=non-kubernetes, and windows.service.mode=generic-host.
• Startup options validation requires agent identity, host identity, install id, environment, positive intervals, bounded spool settings, and at least one local file or command runtime status source. Unlike the Linux agent, the Windows agent does not register an IRuntimeObservationFilter pipeline (no Tetragon bridge / symbol resolver on Windows).
• Configuration precedence is explicit: local files load first, STELLAOPS_WINDOWS_AGENT_ environment variables override file values, and an optional policy-delivered JSON file is loaded last.
• File-backed and command-backed runtime status sources implement Agent.Core IRuntimeStatusSource; command execution uses ProcessStartInfo.ArgumentList, UseShellExecute=false, timeout bounds, and stdout/stderr byte limits.
• The Windows-only platform guard throws before startup on non-Windows platforms.
• The base emits a local WindowsHostAgentSnapshot containing Agent.Core identity, host identity, configuration, capabilities, health, digest-valid runtime statuses, blocked runtime statuses, source issues, and diagnostics.
• The local spool accepts only HostRuntimeStatus records with valid sha256:<64-hex> digests and enforces capacity and retry-attempt bounds.
• Digest-valid spool entries are converted into Agent.Core RuntimeEvidenceEnvelope records and sent to the configured IRuntimeEvidencePublisher.
• Deterministic reason codes cover permission denied, unavailable runtime source, malformed data, tag-only or malformed digest, stale status, ambiguous status, command timeout, over-large output, publish retry, quarantine, drop, and spool retention paths.

Runtime Status Sources

Runtime status sources establish image identity and host/container context. They do not by themselves prove reachability.

Supported direction:

• File-backed local status snapshots for deterministic and air-gapped operation.
• Bounded command-backed local status snapshots from explicitly configured executables.
• Future native Linux runtime APIs only after CRI/runtime contracts, generated types, dependency license review, and fixtures are approved.
• Future native Windows runtime APIs only after API ownership, permission model, and fixtures are approved.

The binding source matrix is maintained in Runtime Status Source Matrix. New source implementations must update that matrix before being marked active.

Rejected direction:

• Inferring image digests from container ids.
• Treating tags as immutable evidence.
• Querying cluster APIs as the RuntimeInstrumentation product model.
• Adding Kubernetes watches, informers, charts, or Helm packaging.

Digest-Gated Evidence Publication

Runtime evidence must be keyed by immutable image digest. The publication path must reject or quarantine observations when:

• No valid sha256:<64-hex> digest is available.
• The digest came only from a tag, container id, or malformed runtime value.
• The source is not enabled, trusted, or configured.
• The evidence envelope cannot be signed or persisted according to the configured mode.
• The local spool is over retention limits.

Evidence publication should include deterministic event ids, source identity, host identity, agent version, runtime status version, observation time, sequence number, digest, and verification metadata so downstream services can replay and audit release decisions.

Current RuntimeInstrumentation implementation:

• Linux and Windows agents build envelopes from local snapshot data only after the runtime status digest gate passes.
• Linux uses matching local RuntimeObservation records when available; otherwise both Linux and Windows create a deterministic RuntimeStatusChanged observation from the runtime status snapshot.
• Duplicate observations within a local snapshot are collapsed before publication.
• AgentCoreValidation.Validate(RuntimeEvidenceEnvelope) rejects malformed evidence: missing event/source ids, non-positive sequence, missing-or-malformed digest (evidence.digest.missing_or_malformed), missing capture timestamp, missing verification metadata (evidence.verification.missing), a digest gate not recorded as passed (evidence.digest_gate.not_passed), a missing attestation mode (evidence.attestation.missing), explicitly unsigned attestation (evidence.attestation.unsigned), digest mismatch between envelope and runtime status (evidence.digest.mismatch), and observation/agent id mismatch (evidence.agent.mismatch). No downstream Signals, Scanner, Policy, EvidenceLocker, or Attestor verification contract exists in RuntimeInstrumentation scope yet, so cryptographic service verification remains a cross-module integration boundary.
• The default NoopRuntimeEvidencePublisher (a singleton registered by both the Linux and Windows agents) returns a retry disposition with reason agent_core.noop_publisher.not_configured to preserve bounded local retention when no downstream publisher is configured.
• Publication last-reason codes surfaced in the local snapshot include linux.spool.full, linux.spool.quarantined, linux.evidence.none, linux.evidence.duplicate, and linux.evidence.publish_exception (and the Windows-agent equivalents).

Optional Tetragon Bridge

The Tetragon bridge is a Linux-only sensor bridge. Its job is to translate local Tetragon output into Agent.Core observation contracts and then let the Linux agent apply the normal digest gate and publisher rules.

Current implementation lives in src/RuntimeInstrumentation/StellaOps.Agent.Tetragon/ and is wired into the Linux agent only when RuntimeInstrumentation:LinuxAgent:TetragonBridge:Enabled=true (bridge options under RuntimeInstrumentation:LinuxAgent:TetragonBridge, export-client options under their own section). Disabled or absent Tetragon leaves the Linux agent with zero optional observation sources and does not block startup. The bridge (TetragonRuntimeObservationSource) reads local Tetragon export output through a named HttpClient (StellaOps.Agent.Tetragon) with bounded reads, maps supported events into Agent.Core RuntimeObservation, and reports observed/dropped/malformed/disconnect counters — plus a TruncatedStackFrames counter — through ITetragonBridgeHealth/TetragonBridgeHealthSnapshot. It never publishes downstream.

Event normalization (TetragonRuntimeObservationAdapter) hoists the canonical process_kprobe wrapper and maps events to a RuntimeObservationKind. Non-kprobe events map by event type (process_exec → ProcessStarted, process_exit → ProcessExited, tcp_connect/network_connection → NetworkConnection, file_access/file_open → FileAccess); kprobe events map by function_name (PHASE-C: e.g. sys_execve → ProcessStarted, tcp_connect → NetworkConnection, do_filp_open → FileAccess, with unknown hooks defaulting to ProcessStarted). Events missing a container id are dropped (tetragon_bridge.container_missing); over-large events are dropped (tetragon_bridge.event_too_large). For kprobe events the adapter extracts stack_trace_user/stack_trace_kernel frames (PHASE-B), truncating each list to 256 frames and surfacing the overflow count, and derives the StackCaptureMode from resolved-symbol coverage. Tetragon emits addresses-only frames; symbol resolution is applied downstream.

Symbol upgrade (Sprint 022 PHASE-D): when the bridge is enabled, AddStellaOpsLinuxSymbolResolver registers the file-backed IProcessSymbolResolver (ProcessSymbolResolver) and RuntimeObservationSymbolUpgrader, and a SymbolResolutionObservationFilter is registered into the Linux agent’s IRuntimeObservationFilter pipeline. The resolver walks /proc/<pid>/maps and the matching ELF symbol table (StellaOps.Agent.Linux.SymbolResolver, with ProcMapsParser, ElfSymbolTableReader, and DwarfLineReader internals) to upgrade addresses-only frames to partial-symbols/full-symbols. It is best-effort and failure-tolerant (a failed lookup returns the input frame unchanged; Line is always null pending DWARF support). Bounds are configured via RuntimeInstrumentation:LinuxAgent:SymbolResolver (ProcessSymbolResolverOptions): 100 ms per-frame timeout, 256 frames per call, 64 cached ELF files, 10 MB cached symbol data, 256 cached process maps, ProcRoot default /proc. No Tetragon means no resolver wiring (disabled-by-default invariant).

Implementation rules:

• It must not register as a standalone Stella Ops host agent.
• It must not be required for Linux agent installation.
• It must not publish runtime evidence directly to downstream services.
• It must not use Kubernetes deployment assumptions.
• It must expose clear health and dropped-event metrics to the Linux agent.
• It must use approved transport contracts, bounded reads, backpressure behavior, and deterministic fixtures.

Tests and CI

Required test layers:

• Agent.Core contract tests for serialization, validation, idempotent event ids, and replay metadata.
• Linux agent unit tests for configuration, runtime status parsing, digest gating, bounded spool behavior, retry behavior, unsupported-platform behavior, health/status snapshots, and absent optional sensors.
• Windows agent unit tests for configuration, runtime status parsing, digest gating, buffering, and publisher retry behavior.
• Optional Linux integration tests on a pinned Linux runner for service packaging, local runtime command execution, and optional Tetragon bridge behavior.
• Optional Windows integration tests on a pinned Windows runner for service hosting, Windows runtime status sources, ACL behavior, and publisher retry behavior.
• Negative tests proving tag-only references, container ids, stale status, malformed status, missing permissions, failed commands, and absent Tetragon do not publish evidence.

CI must not require Kubernetes or Helm for RuntimeInstrumentation verification.

References

• Module README
• Host Agent Strategy
• Linux Host Agent
• Windows Host Agent
• Tetragon Agent Normalization
• Runtime Status Source Matrix