Reachability Runtime Ingestion Runbook

This runbook guides operators through ingesting runtime reachability evidence (EntryTrace, probes, Signals ingestion) and wiring it into the reachability evidence chain, both online and in air-gapped installations. It is written for operators of the Signals service and assumes the runtime-facts schema referenced under Prerequisites.

Source of truth. The endpoint, auth, payload, and storage facts below are reconciled against src/Signals/StellaOps.Signals (Program.cs, Services/RuntimeFactsIngestionService.cs, Storage/*RuntimeFactsArtifactStore.cs, Routing/SignalsPolicies.cs) and the CLI in src/Cli/StellaOps.Cli/Commands (SignalsCommandGroup.cs, ReachabilityCommandGroup.cs). When this doc and src/ disagree, the code wins. Several legacy steps in earlier revisions referenced endpoints, headers, events, and CLI commands that do not exist in src/; those are corrected or flagged inline below.

Imposed rule (policy intent — partially aspirational vs. current code): runtime traces must never bypass CAS/DSSE verification; ingest only CAS-addressed NDJSON with hashes logged to Timeline and Evidence Locker.

Reality check against src/Signals: the operator-facing JSON/NDJSON ingest endpoints (§2 step 4) persist per-fact evidence to PostgreSQL (reachability_facts); they do not CAS-address the body, do not require DSSE on the trace, and do not emit Timeline events. CAS addressing (cas://reachability/runtime-facts/<hash>) and BLAKE3 verification exist only in the service-internal batch store (IngestBatchAsync, no HTTP surface — §2 step 2), and DSSE chains are produced/verified separately via the eBPF evidence pipeline + stella config signals verify-chain. Treat this rule as the target posture; the inline §4 “NOT IMPLEMENTED” flag tracks the missing Timeline wiring.

1. Prerequisites

2. Ingestion workflow (online)

  1. Capture traces from Observer/probes → NDJSON with one runtime-fact event per line. Each event carries symbolId (required) plus optional codeId, symbolDigest, purl, buildId, loaderBase, processId, processName, socketAddress, containerId, evidenceUri, hitCount (default 1), observedAt, and metadata{}. (Model: RuntimeFactEvent in Models/RuntimeFactsIngestRequest.cs.)
  2. Stage to CAS (batch path — service-internal only): the batch artifact store computes a BLAKE3-256 digest of the uploaded NDJSON, stores it immutably, and returns cas://reachability/runtime-facts/<blake3-hex>. (RuntimeFactsIngestionService.IngestBatchAsync + RustFs/FileSystemRuntimeFactsArtifactStore.) Batch ingest refuses to run unless both a crypto-hash provider and the CAS artifact store are configured.

    Flag (NO HTTP SURFACE): IngestBatchAsync is not wired to any route in Program.cs — there is no POST /signals/runtime-facts/batch (or equivalent) endpoint. The batch CAS path is a service-internal capability exercised by RuntimeFactsBatchIngestionTests, not an operator-driven ingest path. Operators ingest runtime facts via the JSON or NDJSON endpoints in step 4 (which do not produce a batch casUri/batchHash). Treat the batch CAS layout in §5 as a description of the internal store, not a runbook step you can drive over the API.

  3. Optionally sign: DSSE signing of runtime evidence chunks is produced by the eBPF evidence pipeline (src/Signals/__Libraries/StellaOps.Signals.Ebpf/Signing/), which emits *.dsse.json sidecars. There is no stella attest runtime CLI command; do not script against one.
  4. Ingest via the Signals API. Two endpoints exist (both require signals:write):
    • Structured JSON — POST /signals/runtime-facts:
      curl -H "Authorization: Bearer <optok>" \
     -H "Content-Type: application/json" \
     --data @runtime-facts.json \
     "https://signals.example/signals/runtime-facts"
      Body: { "subject": { ... }, "callgraphId": "<id>", "events": [ ... ], "metadata": { ... } }.
    • Streaming NDJSON — POST /signals/runtime-facts/ndjson:
      curl -H "Authorization: Bearer <optok>" \
     -H "Content-Type: application/x-ndjson" \
     -H "Content-Encoding: gzip" \
     --data-binary @runtime-trace.ndjson.gz \
     "https://signals.example/signals/runtime-facts/ndjson?callgraphId=<id>&scanId=<scan>"
      The subject is supplied via query parameters (callgraphId required; optional scanId, imageDigest, component, version, purl — see RuntimeFactsStreamMetadata). Gzip is decoded only when Content-Encoding: gzip is set.

    Correction: there is no POST /api/v1/runtime-facts endpoint and no graph_hash query parameter — the keying field is callgraphId. The /api/v1/signals group is read-only (GET /api/v1/signals, GET /api/v1/signals/stats); both accept signals:read or orch:read, and (per CompatibilityApiV1Endpoints.cs) they serve a fixed five-row static fixture (sig-001…sig-005), not live runtime-fact data. Ingest is not tenant-header driven: handlers do not read X-StellaOps-TenantId or the resolved tenant context (see the comment block in Program.cs). Tenant attribution for emitted events comes from request metadata: the signals.fact.updated.v1 envelope resolves tenant from metadata.tenant then metadata.subject.tenant then the configured default (ReachabilityFactEventBuilder.ResolveTenant), while the runtime.updated event resolves it from metadata.tenant_id then "default" (RuntimeFactsIngestionService.EmitRuntimeUpdatedEventAsync).

  5. Read the response. On success the endpoints return 202 Accepted with a Location of /signals/runtime-facts/<subjectKey> and a JSON body (RuntimeFactsIngestResponse): factId, subjectKey, callgraphId, runtimeFactCount, totalHitCount, storedAt.

    Correction: the response does not carry Content-SHA256, X-Graph-Hash, or X-Ingest-Id headers — those do not exist in the handler. Use the JSON body fields for downstream correlation. The casUri/batchHash fields belong to RuntimeFactsBatchIngestResponse, which is returned only by the service-internal IngestBatchAsync (no HTTP endpoint — see step 2 flag); the operator-facing JSON/NDJSON endpoints never return them.

  6. Side effects on ingest (RuntimeFactsIngestionService.IngestAsync): events are aggregated per (symbolId, codeId, loaderBase, purl, symbolDigest, buildId) key (summing hitCount); the fact document is upserted to reachability_facts, the Valkey reachability cache is refreshed, a signals.fact.updated.v1 envelope is published, a runtime.updated event is emitted for policy reanalysis, and reachability is recomputed.
  7. Verify the stored fact: GET /signals/facts/{subjectKey} (requires signals:read). To verify a signed DSSE evidence chain on disk, use stella config signals verify-chain <evidence-dir> --offline (see §5/§7).

3. Ingestion workflow (air-gap)

  1. Receive a runtime bundle containing the NDJSON trace, a manifest with hashes, and optional *.dsse.json evidence chunks.
  2. Validate hashes against the manifest; if DSSE chunks are present, verify the chain structure (see step 4).
  3. Import the trace through the same POST /signals/runtime-facts/ndjson endpoint while the installation is sealed. Sealed-mode enforcement (Signals:AirGap:SealedMode) gates every ingest endpoint with 503 until valid evidence is present, so confirm the seal is healthy first (GET /signals/status reports sealedMode.compliant).
  4. Verify the DSSE evidence chain offline:
    stella config signals verify-chain /path/to/evidence-dir --offline
    This walks *.dsse.json sidecars, checks chunk linkage/sequence/time-monotonicity and signature presence, and exits non-zero on any failure (SignalsCommandGroup.BuildVerifyChainCommand). --offline skips Rekor lookups.

    Correction: there is no signals-offline ingest-runtime binary and no stella graph verify --runtime ... command. stella reachability graph verify exists only as a stub that returns “requires Attestor signature verification” with a not-implemented exit code (ReachabilityCommandGroup.BuildGraphCommand). Do not depend on either in offline runbooks.

  5. Export the ingest receipt (the 202 JSON body from the NDJSON endpoint — factId, subjectKey, callgraphId, runtimeFactCount, totalHitCount, storedAt) and add it to the Evidence Locker. (There is no operator-facing batch casUri; the batch CAS path has no HTTP surface — see §2 step 2.)

4. Checks & alerts

5. Storage and CAS layout

6. Troubleshooting

7. Artefact checklist

8. References