Runtime Agents Architecture

Overview

This document describes the runtime observation layer in StellaOps: eBPF-based kernel instrumentation, container lifecycle monitoring, and signal aggregation. Runtime evidence provides the highest-confidence reachability determination for policy decisions — a function actually invoked at runtime is far stronger evidence than a static call-graph edge.

Altitude note. This is an architecture overview, not an API or schema reference. Enum members, DTO fields, SQL column lists, kernel-config keys, and per-component signatures live in source and drift quickly; this doc keeps the shape and the rationale and points to the authoritative source for the details. Where a section was previously specified as concrete code/DDL that does not match src/, it has been corrected to a shape summary — see Status & drift at the end.

Authoritative sources for the details:

• Signals module: src/Signals/ (dossier: Signals Module Architecture, API: Signals OpenAPI)
• Runtime agents / Tetragon: src/RuntimeInstrumentation/
• Runtime schema (PostgreSQL signals.*): src/Signals/__Libraries/StellaOps.Signals.Persistence/Migrations/ (002_runtime_agent_schema.sql … 005_runtime_facts_stack_frames.sql)
• Auth scopes: signals:read / signals:write / signals:admin in src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs

Runtime Observation Stack

Evidence flows kernel → user space → storage. The kernel layer (eBPF programs attached via kprobes/tracepoints/uprobes) writes raw events to a per-CPU lock-free ring buffer; user-space collectors poll, resolve symbols, correlate to containers, and persist aggregated facts.

┌─────────────────────────────────────────────────────────────────────┐
│ APPLICATION LAYER                                                     │
│   containers → function calls / syscalls / network I/O                │
└───────────────────────────────┬───────────────────────────────────────┘
                                │
┌───────────────────────────────┼───────────────────────────────────────┐
│ KERNEL SPACE                  │                                       │
│   kprobes ─┐   ┌─ tracepoints  │   (verified eBPF bytecode in the VM)  │
│            └─► eBPF VM ◄───────┘                                       │
│                  │                                                     │
│           per-CPU ring buffer (lock-free)                              │
└───────────────────────────────┼──────────────────────[perf events]────┘
                                │
┌───────────────────────────────┼───────────────────────────────────────┐
│ USER SPACE                    ▼                                       │
│   RuntimeSignalCollector (StellaOps.Signals)                          │
│     • perf-event polling • symbol resolution (DWARF/kallsyms)         │
│     • stack unwinding    • container correlation (cgroup) • batching  │
│   Container observer (lifecycle: create/start/stop, image digest)     │
│   RuntimeSignalNormalizer (dedup, aggregate, map to PURL, SBOM enrich)│
└───────────────────────────────┼───────────────────────────────────────┘
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│ STORAGE LAYER — PostgreSQL signals.* (runtime_facts, runtime_agents,  │
│   callgraphs, reachability_facts, func_nodes, call_edges, …)          │
└─────────────────────────────────────────────────────────────────────┘

Key invariants:

• Container correlation is by cgroup id, captured in-kernel and resolved to container_id / image_digest in user space. Without it, runtime events cannot be attributed to a scanned image.
• Symbol identity must match static analysis so runtime hits can be joined to static call-graph nodes (see frame canonicalization below).
• Aggregation is windowed, not per-event; raw event volume (millions/sec) is reduced before persistence.

eBPF Probe Types

StellaOps uses four eBPF attachment mechanisms, each with a distinct purpose and trade-off:

The exact syscall list, tracepoint names, and example probe targets are an implementation detail of the eBPF programs and the dynamic-attachment logic — see the eBPF program sources and probe-selection code under src/RuntimeInstrumentation/ and src/Signals/. Do not treat any fixed list here as authoritative; verify against source.

Event data structures (shape)

The kernel programs emit fixed-layout C structs over the ring buffer, then user space maps them to managed models. Three event families:

• Function-call event — timestamp, pid/tid, cgroup id, function name/address, caller address, stack id, latency, entry/exit/error discriminator.
• Syscall event — timestamp, pid/tid, cgroup id, syscall number, args, return value, latency, stack id.
• Network event — timestamp, pid, cgroup id, protocol/direction, src/dst address+port, byte count, TCP state.

Full field lists and on-wire layout live with the eBPF program definitions and their managed counterparts; verify against src/. The load-bearing invariant across all three: cgroup id is always present (for container correlation) and a monotonic kernel timestamp anchors ordering.

Container Lifecycle Observer (Zastava)

A container-lifecycle observer subscribes to the container runtime (Docker / containerd / CRI-O sockets) and emits lifecycle events (create / start / stop / die / image-pull). On each event it extracts the image reference and digest, labels/annotations, cgroup path, and exit metadata, so runtime facts can be attributed to a specific scanned image digest.

The lifecycle event carries (shape): event type, container id, image ref + sha256: digest, labels/annotations maps, filtered env, pid, cgroup path, start/stop timestamps, exit code, OOM flag.

Source note. Zastava is integrated as a Scanner runtime source and CLI surface (src/Scanner/__Libraries/StellaOps.Scanner.Sources/Handlers/Zastava/, src/Cli/.../ZastavaCommandGroup.cs); the standalone ZastavaObserver class depicted in earlier revisions is conceptual. See the Zastava dossier for the authoritative component model.

Runtime posture

On container start the observer selects an observation posture that determines how much instrumentation is applied. Postures form a ladder from no observation up to full instrumentation, gated by kernel capabilities (CAP_BPF / CAP_SYS_ADMIN), available probe types, container annotations (stellaops.io/observe-level), and debug-symbol availability. Higher postures add syscall+network, then function calls (uprobes), then USDT + coverage. The exact posture names/levels are defined in source — verify against the Zastava/Signals posture model rather than relying on a fixed table here.

Signal Processing Pipeline

Raw events (millions/sec) are reduced to durable facts (per function per window) through four stages:

1. Filtering (in-kernel, eBPF maps) — drop events not from monitored cgroups, keep only security-relevant syscalls, rate-limit per container, sample high-frequency events. This is what keeps userspace volume tractable (millions/sec → thousands/sec).
2. Enrichment (user space) — resolve func_addr → symbol via /proc/<pid>/maps, the SBOM symbol-table cache, DWARF, and kallsyms, yielding func_name + module + purl; correlate cgroup_id → container_id / image_digest / scan_id via the lifecycle registry.
3. Aggregation (time-window, ~1-minute) — key on (image_digest, purl, function_name); accumulate invocation count, unique callers, latency stats, a sample stack, and first/last-seen timestamps.
4. Persistence — upsert aggregated facts into the signals.* runtime tables (idempotent merge on the natural key; counts accumulate, sample data is preserved).

The funnel ratios (millions → thousands → hundreds → per-function-per-window) are the design intent; treat them as orders of magnitude, not guarantees.

Runtime Evidence Schema (shape)

Runtime observations persist in the signals PostgreSQL schema. The authoritative DDL — table names, columns, indexes, and constraints — lives in the embedded migrations:

src/Signals/__Libraries/StellaOps.Signals.Persistence/Migrations/ (002_runtime_agent_schema.sql … 005_runtime_facts_stack_frames.sql)

Core tables (shape only; see migrations for exact columns/indexes):

• signals.runtime_agents — registered runtime agents (tenant, artifact digest, state, heartbeat).
• signals.runtime_facts — observed function/symbol hits per artifact (canonical symbol id, hit count, contexts, last-seen). This is the “is this symbol live at runtime?” fact table.
• signals.callgraphs, signals.func_nodes, signals.call_edges — canonical call-graph nodes/edges (joined to runtime facts by symbol identity).
• signals.reachability_facts — computed reachability scores per call graph.
• signals.cve_func_hits — CVE-to-observed-function joins (the “vulnerable function actually ran” query).
• signals.agent_heartbeats, signals.agent_commands — agent liveness and command queue.

Drift correction. Earlier revisions documented a signals.hot_symbols table with a fixed CREATE TABLE / index / query set. No such table exists in the migrations — “hot symbols” is a concept (and a field, hot_symbols, on an evidence model in src/Signals/StellaOps.Signals/Models/ExecutionEvidenceModels.cs), not a persisted table with that schema. The runtime-fact authority is signals.runtime_facts (+ the canonical-graph and cve_func_hits tables above). Do not reintroduce hand-written DDL here; link the migration.

Typical query shapes (expressed against the real tables; see Signals query/repository code for exact SQL): “is this vulnerable function observed for this image digest?”, “all observed functions for a package”, and “hot-path ranking by hit count”.

Runtime Evidence → Policy Engine

Runtime observation is consumed by reachability scoring as a high-confidence signal. The mapping and confidence model are owned by the policy/scoring engine — see Policy Engine Data Pipeline and the Signals scoring/unified-score docs (docs/modules/signals/unified-score.md) for the authoritative weights and thresholds.

Conceptual mapping (illustrative — exact states/weights live in the scoring engine, verify against source):

• Function observed at runtime (hit count > 0) → treated as dynamically reachable (high confidence).
• Observed on a confirmed vulnerable flow → live exploit path (highest confidence).
• Package present but function never observed over a long window → potentially reachable / likely unused (lower confidence, asymmetric: long no-observation windows actively reduce confidence).
• No runtime posture available → defer to static analysis (neutral).

The runtime factor feeds the unified confidence score as one weighted input alongside static and provenance evidence. Numeric weights and the exact confidence scalars are tuning parameters — do not cite fixed values from this doc; read the scoring engine and unified-score.md.

Evidence chain (shape)

When runtime observation is available it is emitted as a signed evidence record sourced from the Signals runtime facts: a source, an evidence_type of runtime observation, a data payload (image digest, PURL, function, hit count, observation window, sample stack), a confidence scalar, and a timestamp. The exact envelope is defined by the evidence/attestation contracts — see the Signals evidence models (src/Signals/StellaOps.Signals/Models/) and the Attestor/Evidence contracts.

Deployment Considerations

Runtime instrumentation has hard host requirements. Treat the specifics below as orientation and verify exact values against the deployment manifests and the agent’s capability probe at startup.

• Kernel: modern Linux with eBPF + ring-buffer support (BTF/CO-RE strongly recommended for portable programs); uprobe events required for user-space function tracing. The agent probes kernel capabilities at startup and degrades posture if they are absent.
• Privileges: the agent needs eBPF-loading and tracing capabilities (broadly: BPF/PERFMON, SYS_ADMIN, SYS_PTRACE for unwinding, SYS_RESOURCE for memlock, NET_ADMIN for network probes) and mounts for debugfs, the container runtime socket, and /proc. Exact securityContext / capability set lives in the agent DaemonSet manifest — do not hand-maintain the list here.
• Performance: per-probe overhead is small but real (tracepoints cheapest, uprobes most expensive and call-frequency-dependent). Mitigations are architectural: selective attachment to monitored containers only, in-kernel filtering, sampling, and per-container rate limiting. Concrete latency/CPU/memory targets are KPIs owned by the perf suite — see Tetragon integration below.

Tetragon Integration

Tetragon provides kernel-level security observability via eBPF TracingPolicy CRDs. StellaOps integrates Tetragon as a complementary, CNCF-standard runtime observation source (alongside the native Signals agent), well suited to Kubernetes estates with compliance requirements.

Architecture

A Tetragon DaemonSet enforces a TracingPolicy (kprobe/tracepoint/uprobe) and exposes an Export API (gRPC/HTTP). A StellaOps Tetragon agent consumes that stream and runs it through: a privacy filter → an event adapter (Tetragon event → canonical runtime call event) → a frame canonicalizer (Build-ID resolution, C++/Rust/Go demangling, function-ID computation matching static analysis), then fans out to two sinks — the runtime-facts index (PostgreSQL signals.*) and a runtime-witness generator (signing pipeline).

The agent’s export client lives in src/RuntimeInstrumentation/StellaOps.Agent.Tetragon/ (TetragonExportClient, TetragonRuntimeObservationBridge).

Component responsibilities (shape)

The integration decomposes into roles: an agent capability (connects to the Export API, start/stop/status/flush, health checks), an event adapter (Tetragon → canonical call event + process/container context), a frame canonicalizer (buildid:function+offset identity matching static analysis), a hot-symbol bridge (windowed aggregation into the runtime-facts index with confidence scoring), a witness bridge (buffer by claim id → witness generator, with backpressure), and a privacy filter (argument redaction, symbol-id-only mode, namespace allowlisting).

Status note. These roles describe the intended decomposition. As of this writing the live code is the export client + observation bridge above; an IAgentCapability is not registered from StellaOps.Agent.Tetragon (a prior TetragonAgentCapability was removed — see that project’s README.md / TASKS.md). Verify which roles are implemented against src/RuntimeInstrumentation/ before relying on any one being live.

TracingPolicy

The StellaOps TracingPolicy captures process execution (with args), network connections, file operations, and kernel/user stack traces, selected by namespace and by pod label (stellaops.io/observe=true). Authoritative policy manifest: devops/agents-targets/tetragon/tracing-policy-stellaops-default.yaml.

Native Signals vs Tetragon

Performance & operations

KPI targets (CPU/memory overhead, capture latency, throughput, canonicalization/demangling rates) are tracked as test/benchmark assertions in the Tetragon test project under src/RuntimeInstrumentation/StellaOps.RuntimeInstrumentation.Tetragon.Tests/ — read the current benchmark/test source for the live numbers rather than citing fixed targets here. Deployment (Helm install, applying the TracingPolicy, deploying the agent DaemonSet), the agent ConfigMap (Tetragon address, aggregation window, privacy settings, allowed namespaces), and the Prometheus metrics it exposes are operational details owned by the manifests under devops/agents-targets/tetragon/ and the agent project — see those for exact commands, keys, and metric names.

Status & drift (verify before relying)

This doc was right-sized from a code-mirroring revision. Flagged divergences corrected above:

• signals.hot_symbols table — does not exist as documented; the runtime-fact authority is signals.runtime_facts (+ canonical-graph / cve_func_hits tables). “hot_symbols” survives only as a concept and an evidence-model field.
• TracingPolicy path — authoritative manifest is devops/agents-targets/tetragon/tracing-policy-stellaops-default.yaml (the previously-cited devops/manifests/tetragon/… path does not exist).
• Tetragon component classes — most named classes (TetragonAgentCapability, TetragonEventAdapter, etc.) are not present as live registered components; the shipped surface is the export client + observation bridge. TetragonAgentCapability was explicitly removed.
• Benchmark pointer — a fixed TetragonPerformanceBenchmarks.cs path was removed; read the current Tetragon test project for live KPIs.

When this doc and src/ disagree, src/ wins — verify schema, scopes, probe lists, and component wiring against source before relying on any specific detail.

Related Documentation

• Policy Engine Data Pipeline — how runtime feeds policy
• SBOM Analyzer Inventory — the static analyzers whose call-graph nodes runtime facts join to
• Schema Mapping — where the signals.* runtime tables sit in the storage map
• Reachability Drift Alert Flow — runtime-triggered alerts
• Signals Module Architecture — Signals dossier (+ OpenAPI, unified score)
• Zastava Architecture — container observer dossier