Linux Host Agent

The Stella Ops Linux host agent is the RuntimeInstrumentation host service for Linux container estates. It shares StellaOps.Agent.Core contracts with the Windows agent and uses optional Linux sensors only as inputs.

Scope

Installation

Run the Linux agent as a supervised host process under the distribution’s service manager. No Stella Ops package or offline-kit artifact for the agent is built in this repository. The agent must run on the host that owns the container runtime status source.

The release package must not install Kubernetes manifests, Helm charts, or DaemonSets for RuntimeInstrumentation.

Configuration

Configuration precedence is:

  1. Local JSON files: /etc/stellaops/linux-agent.json, then linux-agent.json.
  2. Environment variables prefixed with STELLAOPS_LINUX_AGENT_ (__ maps to :).
  3. Optional policy-delivered JSON file referenced by RuntimeInstrumentation:LinuxAgent:PolicyFile.

Important settings:

Runtime status commands must use fully qualified executable paths, typed arguments, no shell invocation, timeout bounds, and output byte limits.

Runtime Status

Publishable runtime status requires an immutable sha256:<64-hex> image digest. Missing, malformed, tag-only, stale, ambiguous, timed-out, permission-denied, and over-large status inputs fail closed before evidence publication.

File-backed and command-backed local sources are the active baseline. Native CRI/containerd/Docker/Podman APIs remain inactive until contracts, fixtures, permissions, dependencies, and tests are approved.

Symbol resolution and source-line attribution

StellaOps.Agent.Linux.SymbolResolver upgrades runtime observations by reading /proc/<pid>/maps plus the backing ELF binary’s .symtab / .dynsym and, best-effort, its uncompressed DWARF 4/5 .debug_line section. When debug-info is present, the resolved frame carries Module + Symbol + Offset + Line; when debug-info is missing, stripped, or stored under SHF_COMPRESSED, the frame degrades gracefully to full-symbols (Line null) or partial-symbols / addresses-only (per StackCaptureModes). Production images that strip debug-info surface full-symbols frames with a null Line rather than failing resolution. The DWARF reader is intentionally scoped to .debug_line only — DIE walking (.debug_info), type tables, and inline-function attribution are out of scope; line attribution depends on the toolchain emitting an uncompressed line-number program (gcc -gdwarf-4/-gdwarf-5 without -gz=zlib).

Optional Tetragon

Tetragon is not the Linux agent. It is an optional Linux eBPF sensor bridge that can provide RuntimeObservation records to the Linux host agent.

Enable it only when the host has a supported local Tetragon export endpoint:

RuntimeInstrumentation:LinuxAgent:TetragonBridge:Enabled=true
Tetragon:ExportClient:Address=http://127.0.0.1:<port>

Enabling the bridge also registers the symbol resolver and its observation filter; with the bridge disabled, neither is wired.

The bridge never publishes evidence directly. The Linux host agent applies the normal runtime digest gate and Agent.Core publisher path.

Transport gap. TetragonExportClient speaks HTTP: it reads /v1/health, /v1/policies, and an NDJSON event stream at /v1/events under Tetragon:ExportClient:Address. The Tetragon server started below listens on gRPC (--server-address), and the module contains no gRPC client. Pointing the export client directly at the Tetragon gRPC port does not produce observations; an HTTP/NDJSON export endpoint in front of Tetragon is required, and none is shipped in this repository.

Installing the Stella-vendored Tetragon OCI image

Stella Ops vendors a license-verified Tetragon image as stellaops/tetragon (Apache-2.0). See devops/release/components.json vendoredUpstream for the pinned version and digest, devops/airgap/tetragon/Dockerfile for the wrapper image, and third-party-licenses/tetragon-apache-2.0.txt for the license. The commands below use <registry>/stellaops/tetragon:<version>; substitute your registry and pin by digest.

Install via docker (privileged; Tetragon needs kernel capabilities for eBPF). Stage the tracing policy in a host-writable directory FIRST so it is mounted read-only at boot (Tetragon discovers policies at start; kill -HUP is NOT sufficient to load a newly-dropped policy):

sudo mkdir -p /etc/tetragon/policies
sudo cp devops/agents-targets/tetragon/tracing-policy-stellaops-default.yaml \
        /etc/tetragon/policies/stellaops-default.yaml

docker run -d --name tetragon \
  --privileged --pid=host --cgroupns=host \
  -p 127.0.0.1:54321:54321 \
  -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf:ro \
  -v /sys/fs/bpf:/sys/fs/bpf \
  -v /etc/tetragon/policies:/etc/tetragon/policies:ro \
  <registry>/stellaops/tetragon:<version> \
  --tracing-policy-dir /etc/tetragon/policies \
  --server-address 0.0.0.0:54321 \
  --disable-kprobe-multi

Notes:

To reload a policy after editing it on disk, restart the container:

docker restart tetragon

Health and policy probe (gRPC via the bundled CLI):

docker exec tetragon tetra --server-address 127.0.0.1:54321 status
#  → Health Status: running

docker exec tetragon tetra --server-address 127.0.0.1:54321 tracingpolicy list
#  → ID  NAME                STATE     ...  SENSORS          MODE
#    1   stellaops-default   enabled   ...  generic_kprobe   enforce

Live event stream (for verification only; do not leave this running in production):

docker exec tetragon tetra --server-address 127.0.0.1:54321 getevents -o json

Minimum kernel: ≥ 5.4 with BTF (verify ls /sys/kernel/btf/vmlinux). Ubuntu 22.04 / 24.04 kernels ship BTF by default; older RHEL 8 stock kernels do not.

Diagnostics

Each collection cycle publishes a local LinuxHostAgentSnapshot with:

Important reason codes:

Tetragon bridge health reason codes include tetragon_bridge.disabled, tetragon_bridge.starting, tetragon_bridge.connected, tetragon_bridge.disconnected, tetragon_bridge.event_malformed, tetragon_bridge.event_too_large, tetragon_bridge.event_unsupported, and tetragon_bridge.container_missing.

Tetragon export endpoint unreachable

Triggered when the Tetragon bridge health reports tetragon_bridge.disconnected. First confirm the transport gap described under Optional Tetragon: the export client needs an HTTP/NDJSON endpoint, not the Tetragon gRPC port. Then walk this list in order:

  1. Is the container up?

    docker ps --filter name=tetragon --format '{{.Names}} {{.Status}}'
docker logs --tail 50 tetragon 2>&1

    If the container has Exited (1), scroll to the LAST level=fatal line in the logs; common failures are below.

  2. Health probe via the bundled CLI (REST will not work — the server is gRPC):

    docker exec tetragon tetra --server-address 127.0.0.1:54321 status

    Expected: Health Status: running. If this hangs > 5 s the gRPC server has not bound; check docker logs tetragon for “Starting gRPC server”.

  3. Port collision. Tetragon listens on :54321 (configurable via --server-address). Verify nothing else is bound:

    sudo ss -tlnp | grep 54321

    If a stale process holds the port, restart the host or stop the offender.

  4. Kernel BTF missing. The container needs /sys/kernel/btf/vmlinux mounted in. Inside the container:

    docker exec tetragon ls /var/lib/tetragon/btf

    Expected: a non-zero file. If empty, your kernel does not ship BTF (typical for stock RHEL 8). Install a BTF-bearing kernel or extract one via pahole.

  5. bpf_multi_kprobe_v61.o fails to load. Symptom in docker logs:

    level=fatal msg="Failed to start tetragon" error="sensor generic_kprobe ...
  failed prog /var/lib/tetragon/bpf_multi_kprobe_v61.o ... 
  load program: invalid argument"

    The kernel rejects Tetragon’s consolidated multi-kprobe program. Re-launch the container with --disable-kprobe-multi (see install section above). Observed on Ubuntu 6.8.0-111 generic.

  6. AppArmor or SELinux denying eBPF. Symptom: permission denied on BPF_PROG_LOAD or BPF_MAP_CREATE in dmesg. The Stella host agent does not carry an AppArmor profile. If your host hardening adds one, it must allow bpf syscalls and /sys/fs/bpf read+write on the Tetragon container. Quick test: sudo aa-status and sudo cat /var/log/audit/audit.log | grep DENIED | tail -20. SELinux: getenforce and ausearch -m AVC -ts recent.

  7. Tracing policy syntax error. Symptom: level=warning msg="Failed to load TracingPolicy" in docker logs. Validate the YAML:

    python3 -c "import yaml; print(yaml.safe_load(open('/etc/tetragon/policies/stellaops-default.yaml')))"

    If you edit the policy in place, restart the container — Tetragon does not hot-reload policies on SIGHUP.

  8. Observations arrive but Signals is not getting facts. That is not a Tetragon problem. Check RuntimeInstrumentation:RuntimeFacts:BaseAddress (empty keeps the no-op publisher), a callgraph id (RuntimeInstrumentation:RuntimeFacts:CallgraphId or CallgraphIdByImageDigest), and the snapshot’s publication last-reason code (runtime_facts.*). Facts are produced only from symbolicated frames. A runtime_facts.rejected reason with status 401 or 403 means the agent’s Authority client is wrong or lacks signals:write. A runtime_facts.callgraph_not_found reason means Signals does not hold the configured callgraph id: upload that callgraph or correct the mapping. The agent quarantines such entries instead of retrying them.

If steps 1-8 all pass but the bridge still reports tetragon_bridge.disconnected, capture docker logs tetragon plus the agent log line for escalation.