Runtime Instrumentation

Host-agent runtime evidence for non-Kubernetes container estates.

Note: Tetragon is an optional Linux eBPF sensor input, not the Stella Ops Linux agent.

Purpose

Runtime Instrumentation defines how Stella Ops host agents collect local runtime evidence from Linux and Windows container hosts, normalize it into platform contracts, and publish it only when the running container can be tied to an immutable image digest.

The active product model is host-agent based:

• The Stella Ops Linux agent runs on Linux container hosts and owns local runtime status, local sensor inputs, health, buffering, and evidence publication.
• The Stella Ops Windows agent runs on Windows container hosts and owns the equivalent Windows service/runtime integration.
• Tetragon can be used by the Linux agent as an optional eBPF sensor bridge where it is available and approved. It is not the Stella Ops Linux agent itself.
• Kubernetes, Helm charts, DaemonSet deployment, Kubernetes API watches, and Kubernetes-first runtime collection are outside active scope.

Quick Links

• Architecture - Active host-agent architecture and evidence flow
• Host Agent Strategy - Product/technical plan for Linux, Windows, and optional Tetragon inputs
• Linux Host Agent - Linux daemon base, configuration, optional Tetragon setup, diagnostics, and failure modes
• Windows Host Agent - Windows service base, configuration, permissions, diagnostics, and failure modes
• Runtime Status Source Matrix - Local source authority, permissions, freshness, and fail-closed behavior
• Tetragon Agent Normalization - Current status of the quarantined Tetragon client slice

Status

Key Features

• Host-agent model: Runtime evidence is collected by Stella Ops host agents installed on Linux and Windows container hosts.
• Shared Agent.Core contracts: Linux and Windows agents share src/RuntimeInstrumentation/StellaOps.Agent.Core/AgentCoreContracts.cs contracts for identity (AgentIdentity, HostIdentity), capabilities (AgentCapability), runtime status (HostRuntimeStatus), observation intake (RuntimeObservation, IRuntimeObservationSource), health (AgentHealth), configuration (AgentConfiguration), deterministic event ids (RuntimeEvidenceIds.CreateEventId → rtevt:<sha256-hex>), validation (AgentCoreValidation), and evidence publishing (RuntimeEvidenceEnvelope, IRuntimeEvidencePublisher). RuntimeObservation carries optional eBPF stack-frame payloads (UserStack, KernelStack, StackCaptureMode) — backwards-compatible nullable fields; when present, the deterministic event-id incorporates a digest of the stack so observations with different stacks produce distinct ids. See Sprint 20260513_022 PHASE-B.
• Wire-stable enums and stack modes: AgentOperatingSystem (Linux=1, Windows=2), AgentCapabilityKind (RuntimeStatus=1 … OfflineSpool=7), RuntimeKind (Docker=1, Containerd=2, Cri=3, Podman=4, WindowsContainers=5, FileSnapshot=6, CommandSnapshot=7), RuntimeObservationKind (ProcessStarted=1, ProcessExited=2, NetworkConnection=3, FileAccess=4, RuntimeStatusChanged=5, SensorHealthChanged=6), AgentHealthState (Healthy=1 … Stopping=5), and RuntimeEvidencePublishDisposition (Published=1, Retried=2, Quarantined=3, Dropped=4). StackCaptureMode is a string (not a CLR enum, so newer producers can add values): none, addresses-only, partial-symbols, full-symbols. AgentCoreValidation enforces stack-payload budgets of 256 frames per (user/kernel) list, 512-char symbols, 1024-char module paths, and a combined 16 KB stack payload.
• Optional Linux Tetragon bridge: The Linux agent may consume Tetragon output as one local sensor input when the host supports it. Tetragon remains optional and Linux-only. The bridge reads the local Tetragon export endpoint over bounded HTTP (/v1/health, /v1/events as NDJSON, /v1/policies; default address http://localhost:54321, configurable under Tetragon:ExportClient) and maps default-TracingPolicy kprobe hooks (sys_execve, tcp_connect, do_filp_open, exit hooks) and process_exec/process_exit/tcp_connect/file_open event types to RuntimeObservationKind values.
• Best-effort Linux symbol resolution: When the Tetragon bridge is enabled, the Linux agent also wires StellaOps.Agent.Linux.SymbolResolver. ProcessSymbolResolver walks /proc/<pid>/maps and the backing ELF symbol/string tables to upgrade addresses-only Tetragon stack frames to partial-symbols/full-symbols (populating Module/Symbol/Offset). It is failure-tolerant by contract — a failed or timed-out resolution returns frames unchanged and never drops the observation. Bounds (config section RuntimeInstrumentation:LinuxAgent:SymbolResolver): 100 ms per-frame timeout, 256 frames per call, an LRU cache of 64 ELF files / 10 MB / 256 /proc/<pid>/maps entries. DWARF source-line (Line) resolution is present in the reader but deferred — Line is always null today.
• Local runtime status sources: Agents use local runtime authority such as configured snapshot files, bounded local commands, and future approved native runtime APIs.
• Truthful image identity: Runtime observations are published only when they resolve to a valid sha256:<64-hex> image digest. Tags and container ids are lookup material only.
• Digest-gated publication: Evidence is dropped or quarantined when image identity, sensor trust, or publication signing cannot be established.
• Fail-closed behavior: Missing, unreadable, unauthorized, malformed, tag-only, ambiguous, timed-out, or over-large local status does not produce runtime evidence.

Current Live Status Boundary

The current production-capable slice is the local status resolver/worker contract (StellaOps.RuntimeInstrumentation.Worker, whose lifecycle host announces start/stop without producing evidence and binds the snapshot sources from the legacy StellaOps.RuntimeInstrumentation.Tetragon project), the shared StellaOps.Agent.Core contract boundary, the Linux host-agent base in src/RuntimeInstrumentation/StellaOps.Agent.Linux/, the optional best-effort symbol resolver in src/RuntimeInstrumentation/StellaOps.Agent.Linux.SymbolResolver/, the Windows host-agent base in src/RuntimeInstrumentation/StellaOps.Agent.Windows/, and the optional Linux Tetragon observation bridge in src/RuntimeInstrumentation/StellaOps.Agent.Tetragon/. The Linux base provides a generic-host daemon entry point, startup options validation, Linux-only platform guard, source/host identity creation, file-backed and command-backed local runtime status sources implementing Agent.Core IRuntimeStatusSource, optional observation-source intake, a digest gate, bounded local spool state, local health/status snapshot publication, and digest-gated Agent.Core evidence envelope publication. The Windows base provides the equivalent Windows Service-style generic-host lifecycle, Windows-only platform guard, file-backed and bounded command-backed runtime status sources, digest gate, bounded local spool state, local health/status snapshot publication, and digest-gated Agent.Core evidence envelope publication.

The Linux and Windows agents now build RuntimeEvidenceEnvelope records from digest-valid local snapshots and publish them through the Agent.Core IRuntimeEvidencePublisher boundary. Envelopes include deterministic event ids, source identity, host identity, agent identity, sequence, capture timestamp, runtime status freshness, digest, verification metadata, and replay metadata. If no downstream publisher is configured, the default no-op publisher returns a retry result so digest-valid entries remain in the bounded local spool instead of being silently drained.

No Signals, Scanner, Policy, EvidenceLocker, or Attestor service contract is defined inside RuntimeInstrumentation yet, and RuntimeInstrumentation registers no OAuth scopes in the canonical catalog (StellaOps.Auth.Abstractions/StellaOpsScopes.cs) because the module has no HTTP/CLI service surface — it is host-agent-local. The local boundary validates malformed and explicitly unsigned runtime evidence with AgentCoreValidation: a publishable envelope requires a positive Sequence, a valid sha256:<64-hex> digest matching the runtime status digest, an agent-id matching the observation, and VerificationMetadata recording digest.gate=passed plus a non-empty attestation.mode that is not unsigned. Downstream cryptographic signing and service-specific verification remain outside this module boundary.

The local status resolver can consume configured snapshot files and bounded local status commands, then resolve image digests without deriving identity from tags or container ids. Agent.Core now provides the OS-neutral contract surface that Linux and Windows host agents use.

Native CRI socket collection is not active yet. It requires owned CRI protobuf/gRPC contracts or an approved runtime API dependency, license review, generated types, and deterministic fixtures before it can become runtime authority.

Historical Kubernetes list/watch source code and docs were created before the product scope was corrected — the RuntimeContainerImageStatusKubernetesApiSource and RuntimeContainerImageStatusKubernetesWatchSource types still live in the older StellaOps.RuntimeInstrumentation.Tetragon project alongside the command/snapshot sources and ContainerImageDigestResolver. Those Kubernetes paths are not active product direction. New RuntimeInstrumentation work must not add Kubernetes deployment support, Helm charts, Kubernetes API informers, Kubernetes watches, or Kubernetes-first agent behavior.

src/RuntimeInstrumentation/StellaOps.Agent.Tetragon/ now contains the optional Linux Tetragon bridge. It is disabled unless RuntimeInstrumentation:LinuxAgent:TetragonBridge:Enabled=true is configured. TetragonExportClient reads local Tetragon export health, policies, and raw NDJSON event lines through bounded HTTP reads; TetragonRuntimeObservationSource normalizes supported local Tetragon events into Agent.Core RuntimeObservation records for the Linux agent snapshot surface. The bridge never publishes downstream runtime evidence and is not the Linux host agent.

Configuration

Agent options bind from layered JSON files plus environment variables (see the Linux/Windows host-agent dossiers for the full schema).

Command-backed runtime status sources require typed arguments, a fully qualified executable path, an explicit timeout, and an output byte cap (MaxOutputBytes, default 1 MiB, hard ceiling 16 MiB).

Dependencies

Upstream

• Local Linux container runtime authority: containerd/CRI, Docker, Podman, or configured local status commands/files.
• Local Windows container runtime authority: Windows container runtime APIs, Docker named pipe, containerd-on-Windows, or configured local status commands/files.
• Optional Linux sensor: Tetragon, when installed and explicitly enabled.

Downstream

• Signals - Consumes runtime observations for runtime signal scoring.
• Scanner - Uses runtime observations for reachability validation.
• Policy - Incorporates runtime evidence into release decisions.
• EvidenceLocker / Attestor - Receives signed or attestable runtime evidence bundles once publication is implemented.

CI And Release Boundary

.gitea/workflows/runtime-instrumentation-host-agents.yml has a single agent-core-and-linux job (runs-on ubuntu-latest) that runs the Agent.Core contract tests, the Linux host-agent tests, and builds plus tests the optional Tetragon bridge. The Windows host-agent unit lane and the pinned host integration lanes (windows-host-agent, linux-host-integration, windows-host-integration) were deleted on 2026-05-14 because no Gitea runner advertises windows-latest or the runtime-instrumentation self-hosted label set; they must be re-added once a Windows or labelled self-hosted runner exists. The workflow triggers on pull_request/push to src/RuntimeInstrumentation/**, docs/modules/runtime-instrumentation/**, the workflow file itself, and on workflow_dispatch.

The release guide records that RuntimeInstrumentation ships as Linux/Windows host agents, not Kubernetes or Helm packaging. Optional sensor and local-model pin changes must be called out in release notes before publication.