Operator Runbook

Overview

This runbook covers operational procedures for the eBPF runtime reachability evidence collection system (StellaOps.Signals.Ebpf).

Implementation status (verify against src/). The collection library (probe loading, NDJSON evidence writer, DSSE/Rekor chunk signing, chain verification, symbol/container/BTF resolution) is implemented in src/Signals/__Libraries/StellaOps.Signals.Ebpf/. The kernel-attach path in CoreProbeLoader is currently a simulated ring buffer (SimulatedRingBuffer) — it does not yet attach real probes via libbpf, so on-host CPU-overhead and ring-buffer-utilization figures are placeholder values (GetCpuOverhead returns a fixed 0.1, Utilization returns 0.1). Treat the live-host collection sections below as the target operational model; the operator-facing surface that exists today is the stella signals CLI (inspect, list, summary, verify-chain) and the in-process collection API.

Collection surfaces (what actually exists)

Auth scopes for the Signals API (StellaOpsScopes.cs): signals:read, signals:write, signals:admin.

FLAG — not implemented. A standalone stellaops-signals daemon / systemd unit, a stella signals start|stop|status|health|stats command set, and a Prometheus /metrics endpoint for the collector are not present in the codebase. The earlier draft of this runbook described those surfaces; they have been removed or annotated as roadmap below. Re-add concrete procedures here only after the corresponding code lands.

Monitoring

Available instrumentation

The Signals service (src/Signals/StellaOps.Signals/) emits OpenTelemetry metrics only for the “unknowns” decay/rescan pipelines, via the meters StellaOps.Signals.Decay and StellaOps.Signals.Rescan:

FLAG — orphaned (roadmap). The eBPF collector does not currently expose Prometheus metrics. The in-process API surfaces equivalent counters as return values from two distinct stats records:

• SignalStatistics (from RuntimeSignalCollector.GetStatisticsAsync): TotalEvents, EventsPerSecond, DroppedEvents, BufferUtilization, CpuOverheadPercent, MemoryUsageBytes, UniqueCallPaths.
• EvidenceCollectionStats (from RuntimeEvidenceCollector.GetStatsAsync): TotalEvents, ProcessedEvents, DroppedEvents, EventsPerSecond, BufferUtilization, CpuOverhead, MemoryUsage (note the shorter field names — CpuOverhead/MemoryUsage, not the *Percent/*Bytes forms — and that it has no UniqueCallPaths but adds ProcessedEvents).

There are no stellaops_signals_events_total, stellaops_signals_ringbuf_usage, stellaops_signals_drops_total, stellaops_signals_enrich_latency_p99, stellaops_signals_chunks_signed, or stellaops_signals_rekor_failures metrics in the codebase. Do not alert on those names.

Statistics that the collection API reports (when collection is running):

• TotalEvents / EventsPerSecond — throughput counters.
• DroppedEvents — incremented by the rate limiter when MaxEventsPerSecond is exceeded (see RuntimeSignalCollector.ProcessEventsAsync).
• BufferUtilization and CpuOverheadPercent — fixed simulated values from the probe loader (SimulatedRingBuffer.Utilization returns 0.1, CoreProbeLoader.GetCpuOverhead returns 0.1; see status note above).
• MemoryUsageBytes — returns the configured ring-buffer size (CoreProbeLoader.GetMemoryUsage → SimulatedRingBuffer.Size, i.e. RuntimeSignalOptions.RingBufferSize), not a measured RSS figure.

Chain-state inspection

A running collector writes per-chain state files (chain-<key>.json) into the configured ChainStateDirectory and signed chunk sidecars (*.dsse.json) alongside NDJSON chunks (evidence-<yyyyMMddHHmmss>-<seq:000000>.ndjson[.gz]). Verify integrity with the CLI:

# Verify a signed evidence chain on disk (looks for *.dsse.json sidecars)
stella config signals verify-chain /var/lib/stellaops/evidence/

# Verbose, offline (skip Rekor), with a JSON report written out
stella config signals verify-chain /var/lib/stellaops/evidence/ --offline --verbose --report report.json

Common Issues

Issue: Probe failed to attach

Context. Applies once the live libbpf attach path is enabled. With the current simulated loader, CoreProbeLoader.LoadAndAttachAsync throws PlatformNotSupportedException on non-Linux, FileNotFoundException when the probe object (function_tracer.bpf.o) is missing from the probe directory, or InvalidOperationException when no PIDs are found for the target container.

Symptoms (target model):

PlatformNotSupportedException: eBPF probes require Linux.
FileNotFoundException: eBPF probe object not found: /usr/share/stellaops/probes/function_tracer.bpf.o

Diagnosis:

# Confirm the probe object is present in one of the search locations
ls -l /usr/share/stellaops/probes/ /opt/stellaops/probes/

# Check BPF kernel config
grep CONFIG_BPF /boot/config-$(uname -r)

# Check BTF availability (collector resolves a BTF source on startup)
ls -l /sys/kernel/btf/vmlinux

# Check seccomp/AppArmor denials
dmesg | grep -iE "bpf|seccomp|apparmor"

Resolution:

1. Ensure the compiled probe object is installed under one of the search paths (/usr/share/stellaops/probes, /opt/stellaops/probes, or <AppContext.BaseDirectory>/probes).
2. Ensure the host process has the capabilities needed to load BPF programs (CAP_BPF, CAP_PERFMON) or run as root.
3. Confirm a BTF source is resolvable — the collector requires one. Precedence (see RuntimeBtfSourceSelector): kernel BTF (/sys/kernel/btf/vmlinux) → configured external vmlinux → split-BTF directories (/var/lib/stellaops/btf/split, /usr/share/stellaops/btf/split, /usr/lib/stellaops/btf/split). If none is found, IsSupported() returns false and collection throws PlatformNotSupportedException with reason no_btf_source_found.

Issue: Events dropped (rate limiting / buffer pressure)

Symptoms:

• DroppedEvents climbs in GetStatisticsAsync output.

Cause. Events are dropped when the per-second rate limiter rejects them (MaxEventsPerSecond), or when the in-memory event queue exceeds its 100,000 cap and old events are trimmed (RuntimeSignalCollector.ProcessEventsAsync). The simulated ring buffer is a bounded channel with DropOldest semantics.

Resolution:

1. Raise MaxEventsPerSecond on RuntimeSignalOptions (default 10000; there is no “unlimited” sentinel in code — set it high if needed).
2. Increase RingBufferSize on RuntimeSignalOptions (default 262144 bytes = 256 KB).
3. Narrow the trace scope with TargetSymbols and/or raise SampleRate (default 1 = every call; 10 = every 10th call) to reduce volume.

Issue: High memory usage

Symptoms:

• Large RSS, OOM pressure on the collector host.

Diagnosis / contributing factors:

• Symbol resolution cache (EnhancedSymbolResolver): per-process LRU with a 5-minute sliding TTL, capped via EbpfEvidenceOptions.SymbolCacheSizeLimit (default 100000).
• In-memory event queue trimmed at 100,000 events.

Resolution:

1. Lower EbpfEvidenceOptions.SymbolCacheSizeLimit.
2. Reduce RuntimeSignalOptions.MaxStackDepth (default 16) to shrink per-event payloads.
3. Disable symbol resolution (RuntimeSignalOptions.ResolveSymbols = false) if address-only evidence is acceptable.

Issue: Symbol resolution failures

Symptoms:

• FunctionCallEvent.Symbol / Library come back null; only the raw addr is present in evidence.

Diagnosis:

# Check if the binary carries symbols
nm /path/to/binary | head

# Check if debuginfo is present (not stripped)
file /path/to/binary | grep -i "not stripped"

Resolution:

1. Install debug symbols for the target binaries/libraries:

   # Debian/Ubuntu
   apt install libc6-dbg
   
   # RHEL/CentOS
   debuginfo-install glibc
   

2. Accept address-only evidence. Unresolved calls still produce a FunctionCallEvent with addr; downstream correlation can still use the call-stack hash.

Issue: Container resolution failures

Symptoms:

• Evidence records have no container_id enrichment, or the cgroup path does not match a known runtime.

Context. CgroupContainerResolver resolves container identity locally from /proc/<pid>/cgroup and /sys/fs/cgroup, matching containerd, Docker, CRI-O, and Podman scope paths. The emitted identity uses the form {runtime}://{id} (e.g. containerd://<64-hex>, docker://<64-hex>, cri-o://<64-hex>, podman://<64-hex>). This is local introspection — it does not require a runtime resolver such as Zastava.

Diagnosis:

# Inspect the cgroup path the resolver parses
cat /proc/<pid>/cgroup

# Confirm the container runtime
docker ps; crictl ps

Resolution:

1. Confirm the cgroup driver/path matches one of the supported scope patterns (the resolver expects a 64-hex container id in the scope name).
2. Image-digest and SBOM-component enrichment (IImageDigestResolver, ISbomComponentProvider, IContainerStateProvider) is optional and, where wired, comes from a runtime state provider such as Zastava. If only digest enrichment is missing (not the container id), check that provider — not the cgroup resolver.

Issue: Evidence chain verification failure

Symptoms (from stella config signals verify-chain):

Evidence Chain Verification
===========================

Path:       /var/lib/stellaops/evidence/
Chunks:     43
Mode:       Online

Results:
  Passed: 42
  Failed: 1

Chain Status: ✗ INVALID

With --verbose, each failing chunk lists its errors, e.g. Chain broken: expected previous_chunk_id=..., got=..., Sequence gap: ..., Time overlap: ..., or No signatures found in envelope.

Diagnosis:

# Detailed report (per-chunk errors) + machine-readable output
stella config signals verify-chain /var/lib/stellaops/evidence/ --verbose --format json

The verifier checks (see SignalsCommandGroup.BuildVerifyChainCommand and EvidenceChunkFinalizer.VerifyChainAsync):

• chain linkage (previous_chunk_id matches the prior chunk’s chunk_id),
• sequence continuity (chunk_sequence increments by 1),
• time monotonicity (no overlap between chunk time ranges),
• presence of DSSE signatures.

Note. In offline mode the CLI verifies structural integrity only; full cryptographic signature verification requires the signing keys and is performed by the signer (IEvidenceChunkSigner.VerifyAsync).

Resolution:

1. Check for missing or out-of-order *.dsse.json chunk sidecars.
2. Check for disk corruption / truncated chunks.
3. If a gap is the result of an intentional restart, document it in the audit trail. The collector reloads chain state from chain-<key>.json on restart (EvidenceChunkFinalizer.LoadChainStateAsync), so a clean restart continues the chain rather than breaking it.

FLAG — orphaned. There is no stella signals reset-chain command. To start a fresh chain, point the collector at a new/empty ChainStateDirectory + OutputDirectory.

Issue: Rekor submission failures

Context. Transparency-log submission is performed by the production signer AttestorEvidenceChunkSigner, which routes through the internal Attestor service (IAttestationSigningService + IRekorClient / RekorBackend) — not by the collector calling a public Rekor endpoint directly. The default Rekor backend URL configured in Attestor is https://rekor.sigstore.dev, but it is configurable (e.g. an internal mirror) for on-prem/air-gap deployments. The local/dev signer (LocalEvidenceChunkSigner) signs the DSSE PAE with an HMAC (via ICryptoHmac; the algorithm follows the active compliance profile — HMAC-SHA256 on world/fips/kcmvp/eidas, HMAC-GOST3411 on gost, HMAC-SM3 on sm) and never submits to Rekor.

Symptoms:

• Signed chunks have a null RekorUuid / RekorLogIndex when submission fails or is disabled.

Diagnosis:

# Check the Attestor service health (it owns Rekor submission)
# (use your deployment's Attestor health/status surface)

Resolution:

1. Verify connectivity from Attestor to the configured Rekor backend.
2. To run without transparency-log submission, set EvidenceChunkFinalizerOptions.SubmitToRekor = false (or use LocalEvidenceChunkSigner / NullEvidenceChunkSigner in non-production). Chunks are still signed (DSSE) and chained; only the Rekor receipt is absent.

FLAG — orphaned. There is no stella signer status command and no stella signals resubmit-pending command in the CLI. key_id: fulcio from the earlier draft is not how signing keys are selected — the key id is the SigningKeyId passed to the finalizer / the Attestor signing service.

Operational Procedures

Procedures that drove a standalone collector daemon (stella signals start|stop, systemctl ... stellaops-signals, stella signals cleanup, export, import, manifest, resubmit-pending, init, reset-chain) described commands that do not exist in the codebase and have been removed. The procedures below use surfaces that exist today.

Procedure: Verify and archive an evidence directory

# 1. Verify chain integrity before archiving
stella config signals verify-chain /var/lib/stellaops/evidence/ --report pre-archive-report.json

# 2. Archive evidence (NDJSON chunks + *.dsse.json sidecars + chain-*.json state)
tar -czvf evidence-$(date +%Y%m%d).tar.gz /var/lib/stellaops/evidence/

# 3. Move to long-term storage (use your own offline/on-prem object store;
#    do not assume a cloud bucket on an air-gapped host)

Procedure: Air-gap probe bundle handling

The air-gap loader (AirGapProbeLoader) consumes a sealed probe bundle (ebpf-probes.bundle, a zip with a manifest.json) or embedded resources, and verifies each probe’s SHA-256 against the manifest before use. Default bundle search locations: <AppContext.BaseDirectory>/ebpf-probes.bundle, /usr/share/stellaops/probes/ebpf-probes.bundle, /opt/stellaops/probes/ebpf-probes.bundle.

# Place the pre-built bundle where the loader looks for it
cp ebpf-probes.bundle /usr/share/stellaops/probes/

# Enable air-gap mode in the host that registers the collector
#   services.AddEbpfRuntimeEvidenceAirGap(...);

AirGapProbeLoader.VerifyIntegrityAsync() recomputes each probe hash and reports mismatches; CreateBundleAsync(...) builds a bundle during CI for offline deployment.

Procedure: Verify an exported evidence chain offline

# On the air-gapped verifier, after transferring the evidence directory:
stella config signals verify-chain --offline /imported/evidence/ --verbose

Offline mode skips Rekor lookups and verifies chain/sequence/time/signature- presence structurally (see the verification issue above for the full check set).

Configuration Reference

The collector is configured in code via EbpfEvidenceOptions (passed to AddEbpfRuntimeEvidence / AddEbpfRuntimeEvidenceAirGap). There is no free-standing signals: YAML config schema in the codebase; the YAML in the earlier draft (ring_buffer_size, max_events_per_second, filters.paths.allowlist, signing.key_id, metrics.port, etc.) did not map to any real option keys. The authoritative option types are listed below.

EbpfEvidenceOptions (DI registration)

NdjsonWriterOptions (rotation / output)

RuntimeSignalOptions (per-collection)

EvidenceChunkFinalizerOptions (signing / chaining)

Evidence record schema

NDJSON evidence records use schema runtime-evidence/v1 (RuntimeEvidenceRecord): ts_ns (ns since boot), src (probe name), pid, tid, cgroup_id, container_id ({runtime}://{id}, enriched post-collection), image_digest (sha256:..., enriched), comm, and a polymorphic event object (file_open, process_exec, tcp_state, net_connect, ssl_op, function_call). Signed chunks are wrapped in an in-toto DSSE statement with predicate type stella.ops/runtime-evidence@v1 and payload type application/vnd.in-toto+json.

Probe sources

The probes/event sources the parser recognises (EventParser, RuntimeEvidenceRecord.Source):

Supported probe types reported by CoreProbeLoader.GetSupportedProbeTypes() on Linux: Uprobe, Uretprobe, and Usdt (when /sys/kernel/debug/tracing/uprobe_events exists).

Support

For issues not covered here:

1. Confirm behaviour against the source of truth: src/Signals/__Libraries/StellaOps.Signals.Ebpf/ and src/Cli/StellaOps.Cli/Commands/SignalsCommandGroup.cs.
2. Capture the output of stella config signals verify-chain <dir> --verbose --format json.
3. Provide kernel and BTF context (uname -a, ls -l /sys/kernel/btf/vmlinux).
4. Provide the relevant EbpfEvidenceOptions / RuntimeSignalOptions configuration (sanitized).

Related

• eBPF Reachability overview — quick start and architecture
• Evidence Schema — NDJSON record shape and determinism invariants
• Probe Reference — per-probe attach points and limitations
• Security Model — threat model, hardening, and compliance mapping