Runtime Linkage Verification - Operational Runbook

Audience: Platform operators, SREs, security engineers Related: Runtime Linkage Guide, Function Map V1 Contract

Implementation status (verify against src/ before relying on any step):

• Function-map generate / verify (offline) — IMPLEMENTED. stella function-map generate|verify and stella observations query exist (src/Cli/StellaOps.Cli/Commands/FunctionMap/FunctionMapCommandGroup.cs, src/Cli/StellaOps.Cli/Commands/Observations/ObservationsCommandGroup.cs). The claim verifier (src/Scanner/__Libraries/StellaOps.Scanner.Reachability/FunctionMap/Verification/ClaimVerifier.cs) and the PostgreSQL observation store (PostgresRuntimeObservationStore, table scanner.runtime_observations) are real.
• Online verification — NOT IMPLEMENTED in the CLI. stella function-map verify only consumes observations from a file via --offline --observations; the online observation-store query path prints "Online observation query not yet implemented. Use --offline with --observations." Drive online verification through the Platform HTTP API instead (see Platform Function-Map API).
• eBPF capture — SCAFFOLD / SIMULATION. src/Signals/__Libraries/StellaOps.Signals.Ebpf/Probes/CoreProbeLoader.cs simulates probe attach (SimulatedRingBuffer, stubbed FDs, fixed 0.1% overhead) and is Linux-gated; it does not yet load real libbpf programs. Treat the eBPF deployment guidance below as the intended target design, not a verified production path.
• CLI --sign on verify — NOT IMPLEMENTED (logs a warning). Signing/attestation is wired only on function-map generate (--sign for DSSE, --attest for Rekor).

Overview

This runbook covers deployment and operation of the runtime linkage verification system. The system declares expected call-paths as a signed function map predicate (https://stella.ops/predicates/function-map/v1), captures runtime function-call observations via eBPF / Tetragon, and verifies the observations against the map to produce a coverage verdict. Ground-truth code lives in src/Scanner/__Libraries/StellaOps.Scanner.Reachability/FunctionMap/ (predicate, generator, verifier, store), src/Signals/__Libraries/StellaOps.Signals.Ebpf/ (capture), and src/Platform/StellaOps.Platform.WebService/Endpoints/FunctionMapEndpoints.cs (HTTP API).

Prerequisites

The capture path is the eBPF collector in StellaOps.Signals.Ebpf. The kernel/capability requirements below are the intended target for a real libbpf deployment; the shipped CoreProbeLoader is a Linux-gated simulation (see status banner) and these requirements are not yet enforced or probed in code. The collector throws PlatformNotSupportedException on non-Linux hosts.

• Linux host (eBPF capture is Linux-only; RuntimeInformation.IsOSPlatform(OSPlatform.Linux) is asserted in CoreProbeLoader)
• Linux kernel 5.8+ recommended for eBPF CO-RE support (intended target; not verified in code)
• CAP_BPF and CAP_PERFMON (or CAP_SYS_ADMIN on older kernels) for the capture process (intended target)
• BTF (BPF Type Format) enabled in kernel config for CO-RE (intended target)
• Pre-compiled CO-RE probe object (function_tracer.bpf.o) available in a probe directory (see below), or a sealed probe bundle for air-gap (AirGapProbeLoader)
• PostgreSQL reachable by the service that owns the scanner.runtime_observations table (for persisted observations)

Deployment

eBPF capture configuration

There is no stella-runtime-agent binary or runtime_agent: YAML schema in the codebase — those were aspirational. The capture surface is the RuntimeSignalOptions record consumed by the eBPF collector (src/Signals/__Libraries/StellaOps.Signals.Ebpf/Services/IRuntimeSignalCollector.cs). Its real, defaulted fields are:

The probe object (function_tracer.bpf.o) is discovered from the first existing directory of /usr/share/stellaops/probes, /opt/stellaops/probes, then <AppContext.BaseDirectory>/probes (CoreProbeLoader.GetDefaultProbeDirectory). For air-gap, AirGapProbeLoader loads probes from embedded resources or a sealed ebpf-probes.bundle with a manifest.json.

Observations are persisted by the PostgreSQL store only (PostgresRuntimeObservationStore, table scanner.runtime_observations, BRIN index on observed_at for efficient time-range pruning). There is no in-memory or Valkey observation store in production code — an in-memory implementation exists solely as a test double. Retention is applied by calling IRuntimeObservationStore.PruneOlderThanAsync(TimeSpan); there is no retention_hours config key.

Probe Selection Guidance

Prioritization:

1. Start with crypto and auth paths (highest security relevance)
2. Add network I/O for service mesh verification
3. Expand to file access for compliance requirements

The supported probe-type identifiers (validated by FunctionMapSchema.ProbeTypes) are: kprobe, kretprobe, uprobe, uretprobe, tracepoint, usdt. Note that the simulated CoreProbeLoader.GetSupportedProbeTypes() currently advertises only uprobe/uretprobe (plus usdt when /sys/kernel/debug/tracing/uprobe_events exists); kprobe-based syscall capture in the table above is intended target behavior.

Resource Overhead

These figures are planning estimates for a real libbpf deployment, not measured values — the current loader reports a fixed simulated 0.1% CPU overhead (CoreProbeLoader.GetCpuOverhead). Validate against your own workload before sizing.

Expected overhead per probe (estimate):

• CPU: ~0.1-0.5% per active uprobe (per-call overhead ~100ns)
• Memory: ~2KB per attached probe + observation buffer
• Disk: ~100 bytes per observation record (NDJSON)

Tunable limits backed by code (RuntimeSignalOptions):

• Ring buffer: RingBufferSize, default 256 KiB (the 64 MB figure previously documented is not a default)
• Event rate cap: MaxEventsPerSecond, default 10000
• Sampling: SampleRate, default 1 (every call)
• Stack depth: MaxStackDepth, default 16

Operator guidance (not enforced by code):

• Retention: prune via PruneOlderThanAsync(TimeSpan) on a schedule you own (e.g. 72h)

Operations

Generating Function Maps

Run generation as part of CI/CD pipeline after SBOM generation. The flags below match function-map generate (--sbom/-s, --service, --hot-functions/-H repeatable, --min-rate, --window, --build-id, --output/-o, --format json|yaml/-f, optional --subject, --static-analysis, --fail-on-unexpected, --sign, --attest):

# In CI after SBOM generation
stella function-map generate \
  --sbom ${BUILD_DIR}/sbom.cdx.json \
  --service ${SERVICE_NAME} \
  --hot-functions "SSL_*" --hot-functions "EVP_*" --hot-functions "PEM_*" \
  --min-rate 0.95 \
  --window 1800 \
  --build-id ${CI_BUILD_ID} \
  --output ${BUILD_DIR}/function-map.json

--hot-functions is a glob over symbol names (e.g. SSL_*, EVP_*), not path-style categories. --min-rate defaults to 0.95 and --window to 1800 seconds (FunctionMapSchema.DefaultMinObservationRate / DefaultWindowSeconds). Add --sign to emit a DSSE envelope and --attest to submit it to Rekor (REKOR_URL, default http://localhost:3000). Store the function map alongside the container image (OCI referrer or artifact registry).

Verification

The CLI function-map verify only supports offline verification against an NDJSON observations file. The online observation-store query path is not implemented in the CLI and prints a warning. For periodic online verification, call the Platform HTTP API (see Platform Function-Map API).

# Offline verification against a captured observations file
stella function-map verify \
  --function-map /etc/stella/function-map.json \
  --observations /var/stella/observations/obs.ndjson \
  --offline \
  --from "$(date -d '1 hour ago' -Iseconds)" \
  --to "$(date -Iseconds)" \
  --format json --output /var/stella/verification/latest.json

Verify flags (per FunctionMapCommandGroup): --function-map/-m (required), --container/-c, --from, --to, --output/-o, --format json|table|md/-f (default table), --strict (fail on any unexpected symbol), --offline, --observations. --sign is accepted but not yet implemented (logs a warning). The command exits 0 when verified and 25 (VerificationFailed) otherwise; see Exit codes.

Monitoring

The metric/alert names below are target design, not emitted telemetry — no observation_rate, unexpected_symbols_count, probe_attach_failures, or observation_buffer_full metric is published by the current code. The equivalent signals exist today only as fields on the verification report object (ClaimVerificationResult.ObservationRate, .UnexpectedSymbols, .Verified, .Evidence). Treat the thresholds as guidance for operators wiring their own alerting on parsed report output until first-class metrics are wired.

Report fields suitable to alert on (parse the --format json verification output):

Alert Configuration (target design)

The following is illustrative; it does not map to shipped metric names. Wire alerting against parsed report output until first-class metrics exist.

alerts:
  - name: "function-map-coverage-low"
    condition: observation_rate < 0.80
    severity: warning
    description: "Function map coverage below 80% for {service}"
    runbook: "Check probe attachment, verify no binary update without map regeneration"

  - name: "function-map-unexpected-calls"
    condition: unexpected_symbols_count > 0
    severity: info
    description: "Unexpected function calls detected in {service}"
    runbook: "Review unexpected symbols, regenerate function map if benign"

Platform Function-Map API

The Platform service exposes a tenant-scoped HTTP surface for managing and verifying function maps (src/Platform/StellaOps.Platform.WebService/Endpoints/FunctionMapEndpoints.cs). Use this for online verification, where the CLI offline-only path is insufficient. All routes are under /api/v1/function-maps, require a tenant (RequireTenant()), and enforce these authorization policies:

Policy names vs. scope claim values: the ASP.NET policy constants in Constants/PlatformPolicies.cs are platform.functionmap.read|write|verify, while the underlying scope strings in Constants/PlatformScopes.cs are functionmap.read|write|verify. Confirm both against source before configuring clients.

The Signals ingestion path (runtime observation submission/read) is gated by the canonical Authority scopes signals:read, signals:write, and signals:admin (src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs). The function-map and observations CLI commands operate on local files and do not themselves require a scope.

Performance Tuning

High-Traffic Services

For services with high call rates on probed functions:

1. Sampling: Reduce capture volume with RuntimeSignalOptions.SampleRate (integer 1-in-N; default 1 = every call). For example SampleRate = 100 samples 1% of calls. There is no sampling_rate: 0.01 fractional YAML key.
2. Rate cap: RuntimeSignalOptions.MaxEventsPerSecond (default 10000) bounds emitted events. Note: aggregated, count-based observations are represented at the record level via ClaimObservation.observation_count; there is no aggregation_window_ms config key in code.
3. Selective probing: Use --hot-functions at generate time to limit the map to critical symbols, which limits what the capture side attaches to.

Large Function Maps

For maps with many expected paths:

1. Categorize paths with ExpectedPath.tags (e.g. crypto, auth, network) — a free-form string list on each path.
2. Mark low-priority or feature-flagged paths with optional: true (ExpectedPath.optional). Optional paths do not count against coverage requirements.
3. Per-tag minimum rates are not supported: CoverageThresholds exposes a single minObservationRate (plus windowSeconds, optional minObservationCount, and failOnUnexpected). There is no per-tag threshold override.

Storage Optimization

For long-term observation storage:

1. Apply retention pruning via the store API IRuntimeObservationStore.PruneOlderThanAsync(TimeSpan.FromHours(72)) (schedule it from your operator/controller; there is no built-in cron). The Postgres store uses a BRIN index on observed_at, so range-deletes of old rows are cheap.
2. Compress archived observations (gzip NDJSON)
3. Use dedicated Postgres partitions by date for query performance

Incident Response

Coverage Dropped After Deployment

1. Check if binary was updated without regenerating the function map
2. Inspect recent observations: stella observations query --summary (the summary view reports record/observation counts, unique symbols/containers/pods, a probe-type breakdown, and top symbols). In online mode this queries IRuntimeObservationStore; if no store is registered the CLI prints "Observation store not available. Use --offline with --observations-file." and returns an empty result — so use --offline --observations-file <ndjson> when no store is wired.
3. Check for symbol changes (ASLR, different build)
4. Regenerate function map from new SBOM and redeploy

Unexpected Symbols Detected

1. Identify the unexpected functions from the verification report
2. Determine if they are:
   • Benign: Dynamic dispatch, plugins, lazy-loaded libraries → add to map
   • Suspicious: Unexpected crypto usage, network calls → escalate to security team
3. If benign, regenerate function map with broader patterns
4. If suspicious, correlate with vulnerability findings and open incident

Probe Attachment Failures

Reminder: the shipped CoreProbeLoader simulates attach and does not call libbpf, so a real “attach failure” surfaces today as either PlatformNotSupportedException (non-Linux host), No processes found for container {id} (empty cgroup//proc scan), or eBPF probe object not found: {path} (missing function_tracer.bpf.o). The kernel/capability checks below apply to the intended libbpf deployment.

1. Confirm the host is Linux (capture throws PlatformNotSupportedException otherwise)
2. Confirm the probe object is present in a probe directory (/usr/share/stellaops/probes, /opt/stellaops/probes, or <base>/probes); a missing function_tracer.bpf.o raises FileNotFoundException
3. Confirm the container resolves to PIDs (cgroup paths docker-<id>.scope / containerd-<id>.scope, else /proc scan)
4. Check kernel version: uname -r (5.8+ recommended for CO-RE — intended target)
5. Verify BTF: ls /sys/kernel/btf/vmlinux (intended target)
6. Check capabilities: capsh --print | grep -i bpf (intended target)
7. Check for SELinux/AppArmor blocking BPF operations

CLI Exit Codes

function-map (FunctionMapExitCodes):

observations (ObservationsExitCodes):

Air-Gap Considerations

For air-gapped environments:

1. Bundle generation (connected side). Sign the map at generate time (--sign produces a DSSE envelope; --attest additionally submits to Rekor on the connected side):

   stella function-map generate --sbom app.cdx.json --service my-service --sign --output fm.json
   # Package with observations
   tar czf linkage-bundle.tgz fm.json observations/*.ndjson
   

   In a sealed environment the eBPF capture side uses AirGapProbeLoader, which loads probes from embedded resources or a sealed ebpf-probes.bundle (manifest.json) rather than compiling at runtime.

2. Transfer via approved media to air-gapped environment.

3. Offline verification (air-gapped side) — this is the supported CLI verification mode:

   stella function-map verify --function-map fm.json --offline --observations obs.ndjson \
     --format json --output report.json
   

4. Result export for compliance reporting. The verification report is written by --output. Note: function-map verify --sign is not yet implemented, and there is no stella attest sign subcommand (the attest group provides build, attach, verify, verify-offline, list, fetch, predicates). To produce a signed/attested artifact today, sign the function map at generate time with --sign/--attest, or attach the report as an OCI attestation via stella attest attach.