eBPF Reachability — Deployment Guide

Audience: operators deploying the Stella Ops eBPF reachability collector and verifying its evidence output.

This guide covers kernel and runtime prerequisites, installation (standard and air-gap), collector configuration via EbpfEvidenceOptions, and evidence verification with stella config signals verify-chain. For the system design behind the collector, see eBPF Architecture; for day-2 operations, see the Operator Runbook.

Status (reconciled against src/Signals). The eBPF reachability collector is partially implemented. The library StellaOps.Signals.Ebpf contains real components — BPF C probe sources, a CO-RE/air-gap probe-loader API, a deterministic NDJSON evidence writer with SHA-256 chunk chaining, and DSSE/Rekor chunk signing via the Attestor (AttestorEvidenceChunkSigner). However:

• The default probe loader (CoreProbeLoader, wired by AddEbpfRuntimeEvidence) is a simulation: it does not attach BPF programs via libbpf and reads from an in-memory SimulatedRingBuffer. CPU/memory/utilization figures it reports are placeholders. Live kernel attach is not yet implemented.
• The .bpf.c probe sources ship as <None> items (build inputs), not embedded resources, and no compiled *.bpf.o objects or probe manifest.json are checked in. The “pre-compiled probe bundle” path in AirGapProbeLoader therefore has nothing to load until probes are compiled and bundled out-of-band.
• The CLI surface is under stella config signals. verify-chain is implemented for signed evidence files; inspect, list, and summary fail closed with exit code 9 until a persisted Signals query path is wired. There is no stella signals start, status, watch, stats, probes …, check-kernel, check-permissions, test-probes, or validate-config command.

Sections below are annotated NOT IMPLEMENTED or Draft / roadmap where the code does not yet back the prose. Treat unannotated kernel/runtime prerequisites as the requirements for the real probe loader once it lands.

Prerequisites

Kernel Requirements

Minimum: Linux 4.14 with eBPF support Recommended: Linux 5.8+ with BTF and ring buffer support

Verify Kernel Configuration

# Check eBPF support
zcat /proc/config.gz 2>/dev/null | grep -E "CONFIG_BPF|CONFIG_DEBUG_INFO_BTF" || \
cat /boot/config-$(uname -r) | grep -E "CONFIG_BPF|CONFIG_DEBUG_INFO_BTF"

# Required settings:
# CONFIG_BPF=y
# CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_JIT=y (recommended)
# CONFIG_DEBUG_INFO_BTF=y (for CO-RE)

Verify BTF Availability

# Check for BTF in kernel
ls -la /sys/kernel/btf/vmlinux

# If missing, check BTFHub or kernel debug packages

Container Runtime

Supported runtimes:

• containerd 1.4+
• Docker 20.10+
• CRI-O 1.20+

Verify cgroup v2 is available (recommended):

mount | grep cgroup2
# Expected: cgroup2 on /sys/fs/cgroup type cgroup2

Permissions

The live probe loader (once implemented) requires elevated privileges. These are the capabilities the CoreProbeLoader will need to load BPF objects, read the ring buffer, and attach uprobes — they are documented here as a forward-looking requirement; the current simulated loader does not exercise them.

Required capabilities:

• CAP_BPF: Load and manage eBPF programs
• CAP_PERFMON: Access performance monitoring (ring buffer)
• CAP_SYS_PTRACE: Attach uprobes to processes

The collector runs inside the Signals service (StellaOps.Signals, container stellaops-signals); there is no standalone stella signals start daemon to launch. Grant the capabilities on whichever host process hosts the eBPF collector, e.g.:

# Grant required capabilities to the host binary that loads the probes
sudo setcap cap_bpf,cap_perfmon,cap_sys_ptrace+ep <collector-binary>

NOT IMPLEMENTED — the standalone sudo stella signals start / capsh … stella signals start invocations previously shown here do not correspond to any CLI command in src/Cli and have been removed. The eBPF collector is hosted by the Signals service via AddEbpfRuntimeEvidence, not started by a CLI verb.

Installation

Standard Installation

# Install StellaOps CLI
curl -fsSL https://stella.ops/install.sh | bash

# Verify installation
stella version
stella signals --help

Air-Gap Installation

Draft / roadmap. The air-gap probe-bundle workflow is designed but not yet wired end-to-end. AirGapProbeLoader knows how to read a ebpf-probes.bundle ZIP (with a manifest.json and SHA-256 per-probe hashes) and AirGapProbeLoader.CreateBundleAsync can produce one, but no compiled probes are checked into the repository and there is no stella bundle create --include-probes flag in src/Cli. The commands below describe the intended shape once probes are compiled and bundled, not a working path.

Intended workflow for disconnected environments:

# (roadmap) Produce an offline bundle on a connected machine, transfer it,
# then install on the air-gapped host. Probe compilation/bundling is currently
# an out-of-band step (see Probes/Bpf/Makefile) — not yet exposed via the CLI.

The bundle is intended to include:

• Pre-compiled eBPF probes for common kernel versions (*.bpf.o)
• A manifest.json describing each probe (name, min kernel, BTF requirement, sha256)
• Split-BTF files for kernels without built-in BTF (resolved from /var/lib/stellaops/btf/split, /usr/share/stellaops/btf/split, or /usr/lib/stellaops/btf/split by RuntimeBtfSourceSelector)

Probe bundle paths searched by AirGapProbeLoader (default): ebpf-probes.bundle alongside the binary, then /usr/share/stellaops/probes/ and /opt/stellaops/probes/.

Pre-Compiled Probes

NOT IMPLEMENTED — stella signals probes list|install|verify do not exist. The stella signals group exposes only inspect, list, summary, and verify-chain (see Verification). Probe sources live under src/Signals/__Libraries/StellaOps.Signals.Ebpf/Probes/Bpf/ and are compiled with the adjacent Makefile; there is no CLI-driven per-kernel probe installer yet.

Configuration

Reconciled. There is no signals.yaml schema with the keys previously shown here. The eBPF collector is configured in code via EbpfEvidenceOptions (and its nested NdjsonWriterOptions, RuntimeEvidenceCollectorOptions, and RuntimeBtfSelectionOptions), passed to AddEbpfRuntimeEvidence(...) in the host’s DI registration. Per-collection runtime behaviour is set via RuntimeSignalOptions on each StartCollectionAsync call. The options below are the real properties and their source-code defaults; bind them through your standard host configuration (env/JSON) however the Signals service exposes them.

Collector options (EbpfEvidenceOptions)

Evidence writer / rotation (NdjsonWriterOptions)

Chunks are written as evidence-<yyyyMMddHHmmss>-<seq:D6>.ndjson (.gz when compression is enabled) and each chunk carries a rolling sha256: content hash used for chain linking (see Verification). The rolling hash is computed with .NET IncrementalHash over SHA-256 (the in-code “BLAKE3-like” comment notwithstanding — the algorithm is SHA-256).

Collector channel (RuntimeEvidenceCollectorOptions)

Per-collection runtime options (RuntimeSignalOptions)

Signing (EvidenceChunkFinalizerOptions)

Chunk signing is performed by EvidenceChunkFinalizer using an IEvidenceChunkSigner. The production signer is AttestorEvidenceChunkSigner (DSSE via the Attestor’s IAttestationSigningService, transparency-log submission via IRekorClient); a LocalEvidenceChunkSigner (HMAC, no Rekor) exists for dev/test.

The predicate type is stella.ops/runtime-evidence@v1 and the DSSE payload type is application/vnd.in-toto+json. There is no key_id: fulcio toggle — keyless vs. key-based signing is determined by the Attestor’s configured signing service, not by a string literal in this config. Likewise there is no cloud “KMS key ARN” option; StellaOps is self-hosted and keys are managed via the platform crypto stack.

Probe selection, path/network filtering, namespace targeting

Draft / roadmap. The probe-toggle, file-path allow/deny, network CIDR allow/deny, namespace targeting, and per-resource cache/limit knobs previously shown here are not bound by EbpfEvidenceOptions or RuntimeSignalOptions. Some of this filtering exists only inside the BPF C programs (e.g. syscall_openat.bpf.c has a path_filters map and syscall_network.bpf.c has a dest_cidr_filters map), but there is no configuration surface wiring those maps from host config yet. Trace scope is currently governed by TargetSymbols, RuntimeTypes, MaxEventsPerSecond, and SampleRate.

Authorization scopes

The Signals API surface is gated by these canonical scopes (from StellaOpsScopes.cs):

• signals:read — read-only access to reachability signals
• signals:write — write reachability signals
• signals:admin — administrative access to reachability signal ingestion

Running the Collector

Reconciled. The eBPF collector is not a standalone daemon. It is hosted by the Signals service (StellaOps.Signals, container stellaops-signals, slot 44, port 8080) and registered via AddEbpfRuntimeEvidence(...). There is no stellaops-signals systemd unit shipped in this repo, no stella signals start CLI verb, and no stellaops/signals-collector image. Deploy the Signals service the same way as every other StellaOps service — via the platform compose/helm stack — and grant the host the capabilities the live probe loader needs.

Reminder: with the default CoreProbeLoader, no kernel probes are actually attached (the ring buffer is simulated). The host-mount and privilege guidance below is what the live loader will require once kernel attach is implemented; it is kept as forward-looking deployment guidance and marked Draft / roadmap.

Required host access (Draft / roadmap — for the live loader)

The live collector will need read access to the host kernel/cgroup/proc surfaces:

• /sys/kernel/btf/vmlinux (or a split-BTF / external vmlinux source — see RuntimeBtfSourceSelector)
• /sys/fs/cgroup (container → PID resolution; configurable via CgroupRoot)
• /proc (fallback PID scanning and symbol resolution; configurable via ProcRoot)
• a writable evidence directory (default /var/lib/stellaops/evidence, configurable via OutputDirectory)

and the CAP_BPF, CAP_PERFMON, CAP_SYS_PTRACE capabilities (or equivalent privileged execution) on the process hosting the collector.

Concrete systemd / Docker / Kubernetes DaemonSet manifests targeting a dedicated stellaops/signals-collector image were previously shown here. They reference an image and unit that do not exist in this repository and have been removed pending a real collector-deployment artifact. Until then, the collector lives inside the Signals service container.

Verification

The registered commands are under stella config signals:

stella config signals inspect <digest|run-id> [--type call|memory|network|file|process] \
  [--from <iso8601>] [--to <iso8601>] [--limit N] [--format table|json]
stella config signals list [--limit N] [--format table|json]
stella config signals summary <digest|run-id> [--format text|json]
stella config signals verify-chain <evidence-dir> [--offline] [--report <path>] [--format text|json]

NOT IMPLEMENTED — stella signals status, stella signals watch, and stella signals stats do not exist. There is no CLI surface that lists attached probes or reports live ring-buffer utilisation (the loader’s GetBufferUtilization, GetCpuOverhead, and GetMemoryUsage are exposed only on the in-process IRuntimeSignalCollector API, and on the default CoreProbeLoader they return simulated values).

stella config signals inspect, list, and summary are not implemented. They return exit code 9, emit no signal rows, and write no output until a persisted evidence query client is wired.

Verify Evidence Files

# List evidence chunks (default output directory)
ls -la /var/lib/stellaops/evidence/

# Verify signed-chunk chain integrity
stella config signals verify-chain /var/lib/stellaops/evidence

verify-chain is real. It scans the target directory for *.dsse.json sidecar envelopes (not the raw .ndjson chunks), decodes each in-toto statement, and checks: chain linkage (previous_chunk_id), sequence continuity (chunk_sequence), time-range monotonicity, and signature presence. With --offline it skips Rekor lookups and verifies structural integrity only.

Note: the CLI’s verify-chain performs structural checks — it confirms signatures are present but does not re-run full cryptographic verification (that requires the signing keys). Cryptographic chain verification is implemented in-process by EvidenceChunkFinalizer.VerifyChainAsync / IEvidenceChunkSigner.VerifyAsync.

Troubleshooting

See operator-runbook.md for detailed troubleshooting procedures.

Quick Checks

NOT IMPLEMENTED — stella signals check-kernel, check-permissions, test-probes, and validate-config do not exist in src/Cli. Until those land, verify the prerequisites manually:

# Check eBPF / BTF support (see "Prerequisites" above)
ls -la /sys/kernel/btf/vmlinux
cat /boot/config-$(uname -r) | grep -E "CONFIG_BPF|CONFIG_DEBUG_INFO_BTF"

# Check the host process has the required capabilities
getpcaps $(pgrep -f StellaOps.Signals | head -1)

# Inspect probe sources / compile probes out-of-band
ls src/Signals/__Libraries/StellaOps.Signals.Ebpf/Probes/Bpf/
# (build with the adjacent Makefile)