Findings Ledger — Runtime Instrumentation Operations

Operator runbook for the runtime trace ingest, aggregation, score, and timeline subsystem.

Sprint: SPRINT_20260429_014_FindingsLedger_runtime_observability Tasks: FL-RUNTIME-OBS-001…006. See also: runtime-instrumentation.md(feature design), runtime-instrumentation-api.md(API), observability.md(module-wide telemetry).

This runbook is the on-call companion for the runtime instrumentation feature. Goal: a first-time on-call engineer must be able to diagnose any issue in <15 minutes using only this runbook + the dashboards + logs.

Runtime instrumentation is off by default. Enable it with findings:ledger:runtime:enabled = true; while it is disabled, ingest returns 202 Accepted without writing rows and runtime read endpoints return 404 NotFound.

1. Architecture overview

  ┌────────────┐   POST /api/v1/findings/{id}/runtime/traces
  │ eBPF /APM  │ ──────────────────────────────────────────────────►
  │  agent     │
  └────────────┘
                    ┌──────────────────────────────────────────┐
                    │  RuntimeTracesEndpoints (HTTP layer)     │
                    │   - validation                           │
                    │   - records ingest_total{rejected_*}     │
                    └────────────────┬─────────────────────────┘
                                     │
                                     ▼
                    ┌──────────────────────────────────────────┐
                    │  RuntimeInstrumentationService           │
                    │   1. validate                            │
                    │   2. redact (@0xADDR -> stripped)        │
                    │   3. INSERT findings.runtime_traces      │
                    │   4. UPSERT findings.runtime_trace_aggs  │
                    │   5. APPEND findings.runtime_timeline_*  │
                    │   6. RECOMPUTE score (in-line, v1)       │
                    └────────────────┬─────────────────────────┘
                                     │ daily
                                     ▼
                    ┌──────────────────────────────────────────┐
                    │  RuntimeTraceRetentionWorker             │
                    │   - DELETE traces older than threshold   │
                    │   - per-tenant override (default 90d)    │
                    └──────────────────────────────────────────┘

All five steps inside the service are wrapped in OpenTelemetry spans (findings.ledger.runtime.trace.ingest -> 5 child spans). All emit metrics under findings_ledger_runtime_*.

2. Metric inventory

Tenant cardinality cap: the metrics class folds all tenants beyond the 1000th distinct value seen into the literal _overflow bucket so that a runaway tenant population cannot DoS the Prometheus exporter. The cap is process-lifetime; tune RuntimeInstrumentationMetrics.TenantCardinalityCap if your deployment has more than 1000 tenants and you need per-tenant visibility for all of them.

3. Capacity numbers (synthetic load)

Synthetic load run by RuntimeInstrumentationCapacityBench.MeasureSyntheticLoad on 2026-04-28 (developer dev box; Null* repositories — no Postgres).

These represent the service-layer ceiling: validation, redaction, aggregation, score recompute. Production numbers will add the Postgres round-trip — typically dominated by a single insert + one upsert + one insert + the recompute query (4 round-trips, target ≤ 50 ms total under default settings).

Storage growth (estimated):

• One trace row ≈ 1.5 KiB on disk (frame JSON dominates).
• 1000 traces/min/tenant × 100 tenants × 60 min = 6 M rows/hr → 9 GB/hour at peak, ~ 216 GB/day if every tenant maxes out.
• The default 90-day retention caps storage at ~ 19 TB worst-case. Most production deployments will have orders of magnitude lower per-tenant ingest; right-size disk by extrapolating from the findings_ledger_runtime_traces_ingest_total{status="accepted"} rate multiplied by 1.5 KiB.

Re-measure with a real Postgres in CI when the integration-test infrastructure exists; update this section.

4. Alerts and response

Each alert below corresponds to a Prometheus rule in devops/telemetry/grafana/findings-ledger/runtime-alerts.yaml. The rules’ runbook_url annotations link directly to the matching subsection here.

Alert: RuntimeIngestErrorRateHigh

5xx + 4xx ingest fraction > 0.5% over 5 minutes.

Top-3 likely causes:

1. Client schema drift after an agent upgrade — payload validation rejects new shapes. Check findings_ledger_runtime_traces_ingest_total{status="rejected_validation"} by tenant; correlate with the agent version emitted in the trace.
2. Postgres unavailable or migrations pending — rejected_persistence spike. Check Postgres health (pg_isready), disk pressure, connection limits, and findings.runtime_traces table existence.
3. Auth misconfiguration after Authority key rotation — rejected_auth spike. Verify the findings.runtime.write policy + scope mapping.

Investigation:

1. Open the Findings Ledger - Runtime Instrumentation dashboard, **Row 1
   • RED**.
2. Filter the Ingest 4xx/5xx breakdown panel by status to identify the dominant rejection mode.
3. Pivot the query by tenant (topk(10, sum by (tenant, status) (rate(...)))) — a single-tenant spike usually means client misconfiguration; a fleet-wide spike means service-side breakage.
4. Pull the most-recent ERROR / WARN log lines: eventId 1105 (Trace dropped) and 1106 (Trace persistence failed) carry the rejection reason and exception type.

Alert: RuntimeIngestLatencyHigh

Ingest P95 > 500 ms over 5 minutes.

Top-3 likely causes:

1. Score recompute slow (most common). Check the Score recompute p95/p99 panel; if score_recompute > 100 ms consistently you have hit the documented async-trigger threshold (see next alert).
2. Aggregate upsert contention — many concurrent ingests for the same (tenant, finding, artifact, vulnerable_function) key cause Postgres to serialise on the unique constraint. Visible as elevated findings_ledger_runtime_aggregate_upsert_duration_seconds{p95} with normal score recompute.
3. Postgres I/O pressure — disk saturated by other workloads. Verify via pg_stat_activity + system-level disk metrics.

Alert: RuntimeScoreRecomputeP95Threshold

Score recompute P95 > 100 ms for 15 minutes. This is the documented trigger to move score recompute to a background worker (sprint 013 deferred this work; sprint 014 ships the metric that fires the trigger).

Action:

1. If sustained per the trigger conditions in ADR-017 Findings Runtime Score Recompute Async-Migration, see the ADR for the chosen remediation path (Channel + BackgroundService as the default; outbox + worker only if multi-instance / durability requirements demand it).
2. Until the async work lands, mitigate by lowering DefaultScoreWindow (RuntimeInstrumentationService.DefaultScoreWindow) from 24h to 1h — reduces the recent-traces query at the cost of less smoothing on the score.

Alert: RuntimeRedactionActivityFlatlined

redacted_addresses_total is 0 for 30 minutes while ingest is active.

Top-3 likely causes:

1. Upstream agent symbol-format change — addresses now arrive in a shape the regex (@0x[hex]+) no longer matches. Verify the agent build and update RuntimeSymbolRedactor.AddressSuffix if needed.
2. All tenants disabled the agent simultaneously — coincidental but possible during a coordinated rollback. Cross-check traces_ingest_total{status="accepted"} by tenant.
3. Redactor regex performance regression — the regex has a 50ms timeout; if it’s hitting it, RuntimeSymbolRedactor.Apply may be silently returning the input. Capture a sample frame from logs and run it through the redactor in isolation.

Alert: RuntimeRetentionWorkerStalled

retention_deleted_total is 0 for 2 days while ingest is active.

Top-3 likely causes:

1. Worker disabled in config (findings:ledger:runtime:retention:Enabled = false).
2. No tenants configured in Tenants or PerTenant — the worker silently sweeps zero tenants. The v1 design requires explicit tenant enumeration to avoid an implicit dependency on tenant directory listing for runtime traces specifically.
3. All trace data already within retention window — possible if the feature is newly deployed or all traces are <90 days old. Verify by running the DELETE manually with EXPLAIN.

Investigation:

1. Check service logs for eventId 1110 (Runtime trace retention swept) and 1111 (Runtime trace retention sweep failed).
2. Inspect configuration: findings:ledger:runtime:retention:*. The enumerated tenant list must be non-empty (or PerTenant must have keys).
3. If the worker has crashed at startup, the BackgroundService host’s crash log will explain why; restart the service.

5. Common operational scenarios

Scenario: Spike in traces_redacted_addresses_total

A rise (the opposite of RuntimeRedactionActivityFlatlined) means the upstream agent is suddenly producing more address-suffixed symbols than before. Common causes:

• Symbol resolution regression in the agent (failed to demangle).
• Changed sysroot in a base image — debuginfo missing.
• A specific tenant rolled out a new agent build.

Action: identify the tenant via topk(5, sum by (tenant) (rate(...))) and ping the tenant owner with the trace-symbol diff.

Scenario: Spike in traces_ingest_total{status="rejected_validation"}

Find the offending tenant: topk(5, sum by (tenant) (rate(findings_ledger_runtime_traces_ingest_total{status="rejected_validation"}[5m]))).

The validation log line (eventId 1105 with reason=validation) carries the field that failed. 99% of the time it is frames empty, artifactDigest missing, or capturedAt zero.

Scenario: Tenant requests a retention override

1. Add a key to findings:ledger:runtime:retention:PerTenant: {"tenant-acme": "30.00:00:00"} (TimeSpan format, days.hh:mm:ss).
2. Add the tenant to findings:ledger:runtime:retention:Tenants if not already present.
3. Reload service config (or restart). The worker reads IOptionsMonitor so a config reload picks up the change at the next sweep.

Scenario: GDPR data-access request

Per design, the runtime instrumentation pipeline:

• Persists: tenant id, finding id, artifact digest, component PURL, container id, redacted symbol/file/line, captured timestamp, aggregate entry-point evidence.
• Redacts: trailing @0xADDR address tokens from symbol and file.
• Does NOT persist: process command line, environment variables, request bodies, source code, register state, memory contents, request metadata, or container name.

For a tenant-scoped data-access request:

SELECT * FROM findings.runtime_traces WHERE tenant_id = $1;
SELECT * FROM findings.runtime_trace_aggregates WHERE tenant_id = $1;
SELECT * FROM findings.runtime_timeline_events WHERE tenant_id = $1;

For a deletion request: lower the per-tenant retention to a small value (e.g. 00:00:01) and let the next worker sweep delete the rows. Note that aggregates and timeline events are kept indefinitely; if they must also go, run the equivalent DELETE manually.

6. Configuration reference

{
  "findings": {
    "ledger": {
      "runtime": {
        "enabled": true,                  // sprint 011 — disable for clean test runs
        "retention": {
          "Enabled": true,
          "SweepInterval": "1.00:00:00",  // 24h
          "StartDelay":    "00:05:00",    // 5m
          "DefaultRetention": "90.00:00:00",  // 90 days
          "PerTenant": {
            "tenant-acme":   "30.00:00:00"
          },
          "Tenants": ["tenant-acme", "tenant-foobar"]
        }
      }
    }
  }
}

6.1 Migration 013 — best-effort has_entry_point backfill

Sprint reference: SPRINT_20260429_023_Findings_runtime_contract_corrections (column add) + SPRINT_20260429_024_FindingsLedger_runtime_hygiene task FL-RUNTIME-HYG-002 (capacity / caveat documentation).

Migration 013 (013_runtime_aggregates_entry_point.sql) adds the runtime_trace_aggregates.has_entry_point column and, in the same transaction, attempts to backfill the flag on existing rows by joining back to runtime_traces and looking for a frame that is BOTH the entry point AND the vulnerable function for the aggregate’s symbol.

Capacity note. On large runtime_traces tables the backfill is dominated by the JSONB EXISTS subqueries — empirically ≈ 1 second per 10,000 matching aggregate rows on a warm cache. Run a row-count estimate before applying the migration:

-- pre-migration: count aggregates that may be backfilled
SELECT count(*)
  FROM findings.runtime_trace_aggregates
 WHERE has_entry_point = FALSE;

-- pre-migration: count source traces available to drive the backfill
SELECT count(*) FROM findings.runtime_traces;

EXPLAIN for the backfill query typically shows the planner using the (tenant_id, finding_id, artifact_digest)-style index on runtime_traces for the inner join, with a function scan over each trace’s frames JSONB inside the existence check:

Update on runtime_trace_aggregates agg
  ->  Hash Semi Join
        Hash Cond: (agg.tenant_id = trace.tenant_id
                AND agg.finding_id = trace.finding_id
                AND agg.artifact_digest = trace.artifact_digest)
        Filter: ((SubPlan: jsonb_array_elements(trace.frames) frame
                  WHERE frame @> '{"isEntryPoint":true}')
             AND (SubPlan: jsonb_array_elements(trace.frames) frame
                  WHERE frame @> '{"isVulnerableFunction":true}'
                    AND frame ->> 'symbol' = agg.vulnerable_function))

The migration is idempotent: the has_entry_point = FALSE guard ensures re-running it touches zero rows after the first pass.

Best-effort caveat. Aggregates whose source runtime_traces rows have already been retention-pruned cannot be backfilled and remain FALSE. This is acceptable per the post-sprint-023 product decision — the runtime score formula’s entryPointPresence component (30% weight, see ADR-016 Findings Runtime Score Formula) will under-weight these aggregates until fresh traces arrive that re-establish the entry-point evidence. The condition is self-healing: the next ingest for the same (tenant, finding, artifact, vulnerable_function) that contains an entry-point frame flips the flag via the upsert path in PostgresRuntimeTraceRepository.UpsertAggregateAsync.

7. Related dashboards and rules

• Dashboard: devops/telemetry/grafana/findings-ledger/runtime-instrumentation.json
• Alert rules: devops/telemetry/grafana/findings-ledger/runtime-alerts.yaml
• Capacity bench: src/Findings/StellaOps.Findings.Ledger.Tests/RuntimeInstrumentationCapacityBench.cs
• Retention worker tests: src/Findings/StellaOps.Findings.Ledger.Tests/RuntimeTraceRetentionWorkerTests.cs

8. Change log