Runtime Instrumentation

Verified against commit 942a18727c (2026-09-26). Re-verify with: dotnet build src/RuntimeInstrumentation/StellaOps.RuntimeInstrumentation.sln -v minimal -m:1 -p:UseSharedCompilation=false && grep -rn "SectionName = " src/RuntimeInstrumentation --include=*.cs

Host-agent runtime evidence for non-Kubernetes container estates.

Note: Tetragon is an optional Linux eBPF sensor input, not the Stella Ops Linux agent.

Purpose

Runtime Instrumentation defines how Stella Ops host agents collect local runtime evidence from Linux and Windows container hosts, normalize it into platform contracts, and publish it only when the running container can be tied to an immutable image digest.

The product model is host-agent based:

• The Stella Ops Linux agent runs on Linux container hosts and owns local runtime status, local sensor inputs, health, buffering, and evidence publication.
• The Stella Ops Windows agent runs on Windows container hosts and owns the equivalent Windows service/runtime integration.
• Tetragon can be used by the Linux agent as an optional eBPF sensor bridge where it is available and approved. It is not the Stella Ops Linux agent itself.
• Kubernetes, Helm charts, DaemonSet deployment, Kubernetes API watches, and Kubernetes-first runtime collection are outside scope.

Quick Links

• Architecture - Host-agent architecture and evidence flow
• Host Agent Strategy - Product/technical plan for Linux, Windows, and optional Tetragon inputs
• Linux Host Agent - Linux daemon base, configuration, optional Tetragon setup, diagnostics, and failure modes
• Windows Host Agent - Windows service base, configuration, permissions, diagnostics, and failure modes
• Runtime Status Source Matrix - Local source authority, permissions, freshness, and fail-closed behavior
• Tetragon Agent Normalization - Optional Tetragon bridge event mapping

Status

Projects

Key Features

• Host-agent model: Runtime evidence is collected by Stella Ops host agents installed on Linux and Windows container hosts.
• Shared Agent.Core contracts: Linux and Windows agents share src/RuntimeInstrumentation/StellaOps.Agent.Core/AgentCoreContracts.cs contracts for identity (AgentIdentity, HostIdentity), capabilities (AgentCapability), runtime status (HostRuntimeStatus), observation intake (RuntimeObservation, IRuntimeObservationSource), health (AgentHealth), configuration (AgentConfiguration), deterministic event ids (RuntimeEvidenceIds.CreateEventId, prefix rtevt:), validation (AgentCoreValidation), and evidence publishing (RuntimeEvidenceEnvelope, IRuntimeEvidencePublisher). RuntimeObservation carries optional stack-frame payloads (UserStack, KernelStack, StackCaptureMode); when present, the deterministic event id incorporates a digest of the stack.
• Wire-stable enums and stack modes: AgentOperatingSystem (Linux=1, Windows=2), AgentCapabilityKind (RuntimeStatus=1 … OfflineSpool=7), RuntimeKind (Docker=1, Containerd=2, Cri=3, Podman=4, WindowsContainers=5, FileSnapshot=6, CommandSnapshot=7), RuntimeObservationKind (ProcessStarted=1, ProcessExited=2, NetworkConnection=3, FileAccess=4, RuntimeStatusChanged=5, SensorHealthChanged=6), AgentHealthState (Healthy=1 … Stopping=5), and RuntimeEvidencePublishDisposition (Published=1, Retried=2, Quarantined=3, Dropped=4). StackCaptureMode is a string (none, addresses-only, partial-symbols, full-symbols) so newer producers can add values. AgentCoreValidation enforces stack-payload budgets of 256 frames per list, 512-char symbols, 1024-char module paths, and a combined 16 KB stack payload.
• Optional Linux Tetragon bridge: The Linux agent may consume Tetragon output as one local sensor input. The export client reads a local HTTP export endpoint (/v1/health, /v1/events as NDJSON, /v1/policies; default address http://localhost:54321, configured under Tetragon:ExportClient). See Linux Host Agent for the transport gap against the stock Tetragon gRPC server.
• Best-effort Linux symbol resolution: When the Tetragon bridge is enabled, the Linux agent also wires StellaOps.Agent.Linux.SymbolResolver. ProcessSymbolResolver walks /proc/<pid>/maps and the backing ELF symbol tables to upgrade addresses-only frames to partial-symbols/full-symbols, and populates Line from an uncompressed DWARF .debug_line section when one is present. A failed or timed-out resolution returns frames unchanged and never drops the observation.
• Local runtime status sources: Agents use configured snapshot files and bounded local commands. Native runtime APIs are not implemented.
• Truthful image identity: Runtime observations are published only when they resolve to a valid sha256:<64-hex> image digest. Tags and container ids are lookup material only.
• Fail-closed behavior: Missing, unreadable, unauthorized, malformed, tag-only, ambiguous, timed-out, or over-large local status does not produce runtime evidence.

Evidence Delivery

The Linux agent delivers runtime facts to the Signals ingest endpoint POST /signals/runtime-facts/ndjson (mapped in src/Signals/StellaOps.Signals/Program.cs) through StellaOps.RuntimeInstrumentation.RuntimeFacts:

• LinuxRuntimeFactsEvidencePublisher replaces the default NoopRuntimeEvidencePublisher only when RuntimeInstrumentation:RuntimeFacts:BaseAddress is configured. An unconfigured host keeps the no-op publisher, which returns a retry result so digest-valid entries stay in the bounded spool.
• The endpoint requires a callgraph id. It comes from CallgraphIdByImageDigest (per image digest) or CallgraphId; without either the envelope is retried with reason runtime_facts.callgraph_id_not_configured.
• Facts are produced only from frames the symbol resolver actually resolved. Status-only envelopes and addresses-only frames yield no facts and are reported as dropped.
• The wire field names, NDJSON framing, and aggregation key mirror the Signals RuntimeFactEvent/RuntimeFactKey contract and are covered by StellaOps.Signals.Tests.RuntimeFactsCollectorContractTests.
• The Linux agent’s publisher client mints a client-credentials token with scope signals:write from RuntimeInstrumentation:RuntimeFacts:Authority and sends it as a Bearer header; the Signals endpoint enforces that scope. A configured BaseAddress without an Authority client fails at startup.
• Open gap: symbolId is contractually a callgraph node id, which a host agent cannot learn, so the Linux publisher sends the resolved native symbol name there. Facts join a static callgraph node only through symbolDigest, and only when a purl is available on the observation. Symbol-to-node correlation is not solved.

The Windows agent has no downstream publisher; it always uses NoopRuntimeEvidencePublisher.

Local Status Worker (withdrawn)

The local status worker is withdrawn. On 2026-09-27 PCD-73 moved StellaOps.RuntimeInstrumentation.Worker, the csproj-less StellaOps.RuntimeInstrumentation.Tetragon/ folder of container-image status snapshot sources and digest resolver that only it compiled, and their tests to src/__Obsoleted/RuntimeInstrumentation/ (owner ruling, CoC 15.4). The worker produced no evidence and nothing consumed its snapshots. Those sources included historical Kubernetes API and watch sources; new work must not add Kubernetes deployment support, Helm charts, Kubernetes API informers, Kubernetes watches, or Kubernetes-first agent behavior.

Configuration

Agent options bind from layered JSON files plus environment variables (see the Linux/Windows host-agent documents for the full schema).

Command-backed runtime status sources require typed arguments, a fully qualified executable path, an explicit timeout, and an output byte cap (MaxOutputBytes, default 1 MiB, hard ceiling 16 MiB).

Dependencies

Upstream

• Local container runtime status exported to snapshot files or produced by configured local commands.
• Optional Linux sensor: Tetragon, when installed and explicitly enabled.

Downstream

• Signals - the only wired consumer: Linux runtime facts through POST /signals/runtime-facts/ndjson. Other modules consume runtime facts from Signals, not from this module.

Build-time

• src/__Libraries/StellaOps.Reachability.Core (referenced by the RuntimeFacts project).

CI And Release Boundary

.gitea/workflows/runtime-instrumentation-host-agents.yml has a single agent-core-and-linux job (ubuntu-latest) that builds the module solution and runs the Agent.Core contract tests, the runtime-facts projection tests, the Linux host-agent tests, and the optional Tetragon bridge build and tests. The Windows host-agent tests are not run in CI because no runner advertises windows-latest. The workflow runs on pull_request/push to the paths it lists (including src/RuntimeInstrumentation/** and docs/modules/runtime-instrumentation/**) and on workflow_dispatch.

RuntimeInstrumentation ships no image or package in the release bundle. Any future packaging is host packaging (service manager / Windows Service), never Kubernetes or Helm packaging.