Linux Host Agent

The Stella Ops Linux host agent is the RuntimeInstrumentation host service for Linux container estates. It shares StellaOps.Agent.Core contracts with the Windows agent and uses optional Linux sensors only as inputs.

Scope

Installation

Install the Linux agent as a supervised host process using the distribution’s service manager or the Stella Ops offline kit packaging for the host. The agent must run on the host that owns the container runtime status source.

The release package must not install Kubernetes manifests, Helm charts, or DaemonSets for RuntimeInstrumentation.

Configuration

Configuration precedence is:

  1. File settings.
  2. Environment variables.
  3. Persisted policy settings.

Important settings:

Runtime status commands must use fully qualified executable paths, typed arguments, no shell invocation, timeout bounds, and output byte limits.

Runtime Status

Publishable runtime status requires an immutable sha256:<64-hex> image digest. Missing, malformed, tag-only, stale, ambiguous, timed-out, permission-denied, and over-large status inputs fail closed before evidence publication.

File-backed and command-backed local sources are the active baseline. Native CRI/containerd/Docker/Podman APIs remain inactive until contracts, fixtures, permissions, dependencies, and tests are approved.

Symbol resolution and source-line attribution

StellaOps.Agent.Linux.SymbolResolver upgrades runtime observations by reading /proc/<pid>/maps plus the backing ELF binary’s .symtab / .dynsym and, best-effort, its uncompressed DWARF 4/5 .debug_line section. When debug-info is present, the resolved frame carries Module + Symbol + Offset + Line; when debug-info is missing, stripped, or stored under SHF_COMPRESSED, the frame degrades gracefully to full-symbols (Line null) or partial-symbols / addresses-only (per StackCaptureModes). Production images that strip debug-info will continue to surface full-symbols-no-line rather than failing resolution. The DWARF reader is intentionally scoped to .debug_line only — DIE walking (.debug_info), type tables, and inline-function attribution are out of scope; line attribution depends on the toolchain emitting an uncompressed line-number program (gcc -gdwarf-4/-gdwarf-5 without -gz=zlib).

Optional Tetragon

Tetragon is not the Linux agent. It is an optional Linux eBPF sensor bridge that can provide RuntimeObservation records to the Linux host agent.

Enable it only when the host has a supported local Tetragon export endpoint:

RuntimeInstrumentation:LinuxAgent:TetragonBridge:Enabled=true
RuntimeInstrumentation:LinuxAgent:TetragonBridge:Endpoint=http://127.0.0.1:<port>

The bridge never publishes evidence directly. The Linux host agent applies the normal runtime digest gate and Agent.Core publisher path.

Installing the Stella-vendored Tetragon OCI image

Stella Ops vendors a license-verified Tetragon binary as stellaops/tetragon:v1.4.0 (Apache-2.0). The image is pushed to the lab registry at 127.1.1.5/stellaops/tetragon:v1.4.0. See devops/release/components.json vendoredUpstream for digest metadata and third-party-licenses/tetragon-apache-2.0.txt for the LICENSE.

Install via docker (privileged; Tetragon needs kernel capabilities for eBPF). Stage the tracing policy in a host-writable directory FIRST so it is mounted read-only at boot (Tetragon discovers policies at start; kill -HUP is NOT sufficient to load a newly-dropped policy):

sudo mkdir -p /etc/tetragon/policies
sudo cp devops/agents-targets/tetragon/tracing-policy-stellaops-default.yaml \
        /etc/tetragon/policies/stellaops-default.yaml

docker run -d --name tetragon \
  --privileged --pid=host --cgroupns=host \
  -p 127.0.0.1:54321:54321 \
  -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf:ro \
  -v /sys/fs/bpf:/sys/fs/bpf \
  -v /etc/tetragon/policies:/etc/tetragon/policies:ro \
  127.1.1.5/stellaops/tetragon:v1.4.0 \
  --tracing-policy-dir /etc/tetragon/policies \
  --server-address 0.0.0.0:54321 \
  --disable-kprobe-multi

Notes:

To reload a policy after editing it on disk, restart the container:

docker restart tetragon

Health and policy probe (gRPC via the bundled CLI):

docker exec tetragon tetra --server-address 127.0.0.1:54321 status
#  → Health Status: running

docker exec tetragon tetra --server-address 127.0.0.1:54321 tracingpolicy list
#  → ID  NAME                STATE     ...  SENSORS          MODE
#    1   stellaops-default   enabled   ...  generic_kprobe   enforce

Live event stream (for verification only — do NOT leave this running in production; the Stella Linux agent’s TetragonRuntimeObservationBridge consumes the gRPC stream directly):

docker exec tetragon tetra --server-address 127.0.0.1:54321 getevents -o json

Minimum kernel: ≥ 5.4 with BTF (verify ls /sys/kernel/btf/vmlinux). Ubuntu 22.04 / 24.04 kernels ship BTF by default; older RHEL 8 stock kernels do not.

Diagnostics

Each collection cycle publishes a local LinuxHostAgentSnapshot with:

Important reason codes:

Tetragon export endpoint unreachable

Triggered when the Stella Linux agent logs tetragon_bridge.dial_failed / tetragon_bridge.stream_dropped, or runtime_status.source_failed referencing the Tetragon bridge. Walk this list in order:

  1. Is the container up?

    docker ps --filter name=tetragon --format '{{.Names}} {{.Status}}'
docker logs --tail 50 tetragon 2>&1

    If the container has Exited (1), scroll to the LAST level=fatal line in the logs; common failures are below.

  2. Health probe via the bundled CLI (REST will not work — the server is gRPC):

    docker exec tetragon tetra --server-address 127.0.0.1:54321 status

    Expected: Health Status: running. If this hangs > 5 s the gRPC server has not bound; check docker logs tetragon for “Starting gRPC server”.

  3. Port collision. Tetragon listens on :54321 (configurable via --server-address). Verify nothing else is bound:

    sudo ss -tlnp | grep 54321

    If a stale process holds the port, restart the host or stop the offender.

  4. Kernel BTF missing. The container needs /sys/kernel/btf/vmlinux mounted in. Inside the container:

    docker exec tetragon ls /var/lib/tetragon/btf

    Expected: a non-zero file. If empty, your kernel does not ship BTF (typical for stock RHEL 8). Install a BTF-bearing kernel or extract one via pahole.

  5. bpf_multi_kprobe_v61.o fails to load. Symptom in docker logs:

    level=fatal msg="Failed to start tetragon" error="sensor generic_kprobe ...
  failed prog /var/lib/tetragon/bpf_multi_kprobe_v61.o ... 
  load program: invalid argument"

    The kernel rejects Tetragon’s consolidated multi-kprobe program. Re-launch the container with --disable-kprobe-multi (see install section above). Observed on Ubuntu 6.8.0-111 generic.

  6. AppArmor or SELinux denying eBPF. Symptom: permission denied on BPF_PROG_LOAD or BPF_MAP_CREATE in dmesg. The Stella host agent does NOT carry an AppArmor profile today (see Sprint 022 R3 finding). If your operator-hardening sprint adds one, it must allow bpf syscalls and /sys/fs/bpf read+write on the Tetragon container. Quick test: sudo aa-status and sudo cat /var/log/audit/audit.log | grep DENIED | tail -20. SELinux: getenforce and ausearch -m AVC -ts recent.

  7. Tracing policy syntax error. Symptom: level=warning msg="Failed to load TracingPolicy" in docker logs. Validate the YAML:

    python3 -c "import yaml; print(yaml.safe_load(open('/etc/tetragon/policies/stellaops-default.yaml')))"

    If you edit the policy in place, restart the container — Tetragon does not hot-reload policies on SIGHUP.

  8. Container can reach Tetragon but Signals isn’t getting facts. That is NOT a Tetragon problem; check the Linux agent’s RuntimeInstrumentation:LinuxAgent:TetragonBridge:Endpoint setting AND Signals:IngestEndpoint in the agent’s appsettings. The agent’s tetragon_bridge.observation_published metric increments on success; tetragon_bridge.observation_dropped_* metrics tag drop reason.

If steps 1-8 all pass but the agent still reports tetragon_bridge.dial_failed, capture docker logs tetragon plus the agent log line and file a ticket — the gRPC handshake is failing for a reason not yet enumerated here.