Findings Ledger — Runtime Instrumentation Operations

Operator runbook for the runtime trace ingest, aggregation, score, and timeline subsystem.

Sprint: SPRINT_20260429_014_FindingsLedger_runtime_observability Tasks: FL-RUNTIME-OBS-001…006. See also: runtime-instrumentation.md(feature design), runtime-instrumentation-api.md(API), observability.md(module-wide telemetry).

This runbook is the on-call companion for the runtime instrumentation feature. Goal: a first-time on-call engineer must be able to diagnose any issue in <15 minutes using only this runbook + the dashboards + logs.

Runtime instrumentation is off by default. Enable it with findings:ledger:runtime:enabled = true; while it is disabled, ingest returns 202 Accepted without writing rows and runtime read endpoints return 404 NotFound.

1. Architecture overview

  ┌────────────┐   POST /api/v1/findings/{id}/runtime/traces
  │ eBPF /APM  │ ──────────────────────────────────────────────────►
  │  agent     │
  └────────────┘
                    ┌──────────────────────────────────────────┐
                    │  RuntimeTracesEndpoints (HTTP layer)     │
                    │   - validation                           │
                    │   - records ingest_total{rejected_*}     │
                    └────────────────┬─────────────────────────┘
                                     │
                                     ▼
                    ┌──────────────────────────────────────────┐
                    │  RuntimeInstrumentationService           │
                    │   1. validate                            │
                    │   2. redact (@0xADDR -> stripped)        │
                    │   3. INSERT findings.runtime_traces      │
                    │   4. UPSERT findings.runtime_trace_aggs  │
                    │   5. APPEND findings.runtime_timeline_*  │
                    │   6. RECOMPUTE score (in-line, v1)       │
                    └────────────────┬─────────────────────────┘
                                     │ daily
                                     ▼
                    ┌──────────────────────────────────────────┐
                    │  RuntimeTraceRetentionWorker             │
                    │   - DELETE traces older than threshold   │
                    │   - per-tenant override (default 90d)    │
                    └──────────────────────────────────────────┘

All five steps inside the service are wrapped in OpenTelemetry spans (findings.ledger.runtime.trace.ingest -> 5 child spans). All emit metrics under findings_ledger_runtime_*.

2. Metric inventory

Tenant cardinality cap: the metrics class folds all tenants beyond the 1000th distinct value seen into the literal _overflow bucket so that a runaway tenant population cannot DoS the Prometheus exporter. The cap is process-lifetime; tune RuntimeInstrumentationMetrics.TenantCardinalityCap if your deployment has more than 1000 tenants and you need per-tenant visibility for all of them.

3. Capacity numbers (synthetic load)

Synthetic load run by RuntimeInstrumentationCapacityBench.MeasureSyntheticLoad on 2026-04-28 (developer dev box; Null* repositories — no Postgres).

These represent the service-layer ceiling: validation, redaction, aggregation, score recompute. Production numbers will add the Postgres round-trip — typically dominated by a single insert + one upsert + one insert + the recompute query (4 round-trips, target ≤ 50 ms total under default settings).

Storage growth (estimated):

• One trace row ≈ 1.5 KiB on disk (frame JSON dominates).
• 1000 traces/min/tenant × 100 tenants × 60 min = 6 M rows/hr → 9 GB/hour at peak, ~ 216 GB/day if every tenant maxes out.
• The default 90-day retention caps storage at ~ 19 TB worst-case. Most production deployments will have orders of magnitude lower per-tenant ingest; right-size disk by extrapolating from the findings_ledger_runtime_traces_ingest_total{status="accepted"} rate multiplied by 1.5 KiB.

Re-measure with a real Postgres in CI when the integration-test infrastructure exists; update this section.

4. Alerts and response

Each alert below corresponds to a Prometheus rule in devops/telemetry/grafana/findings-ledger/runtime-alerts.yaml. The rules’ runbook_url annotations link directly to the matching subsection here.

Alert: RuntimeIngestErrorRateHigh

5xx + 4xx ingest fraction > 0.5% over 5 minutes.

Top-3 likely causes:

1. Client schema drift after an agent upgrade — payload validation rejects new shapes. Check findings_ledger_runtime_traces_ingest_total{status="rejected_validation"} by tenant; correlate with the agent version emitted in the trace.
2. Postgres unavailable or migrations pending — rejected_persistence spike. Check Postgres health (pg_isready), disk pressure, connection limits, and findings.runtime_traces table existence.
3. Auth misconfiguration after Authority key rotation — rejected_auth spike. Verify the findings.runtime.write policy + scope mapping.

Investigation:

1. Open the Findings Ledger - Runtime Instrumentation dashboard, **Row 1
   • RED**.
2. Filter the Ingest 4xx/5xx breakdown panel by status to identify the dominant rejection mode.
3. Pivot the query by tenant (topk(10, sum by (tenant, status) (rate(...)))) — a single-tenant spike usually means client misconfiguration; a fleet-wide spike means service-side breakage.
4. Pull the most-recent ERROR / WARN log lines: eventId 1105 (Trace dropped) and 1106 (Trace persistence failed) carry the rejection reason and exception type.

Alert: RuntimeIngestLatencyHigh

Ingest P95 > 500 ms over 5 minutes.

Top-3 likely causes:

1. Score recompute slow (most common). Check the Score recompute p95/p99 panel; if score_recompute > 100 ms consistently you have hit the documented async-trigger threshold (see next alert).
2. Aggregate upsert contention — many concurrent ingests for the same (tenant, finding, artifact, vulnerable_function) key cause Postgres to serialise on the unique constraint. Visible as elevated findings_ledger_runtime_aggregate_upsert_duration_seconds{p95} with normal score recompute.
3. Postgres I/O pressure — disk saturated by other workloads. Verify via pg_stat_activity + system-level disk metrics.

Alert: RuntimeScoreRecomputeP95Threshold

Score recompute P95 > 100 ms for 15 minutes. This is the documented trigger to move score recompute to a background worker (sprint 013 deferred this work; sprint 014 ships the metric that fires the trigger).

Action:

1. If sustained per the trigger conditions in ADR-017 Findings Runtime Score Recompute Async-Migration, see the ADR for the chosen remediation path (Channel + BackgroundService as the default; outbox + worker only if multi-instance / durability requirements demand it).
2. Until the async work lands, mitigate by lowering DefaultScoreWindow (RuntimeInstrumentationService.DefaultScoreWindow) from 24h to 1h — reduces the recent-traces query at the cost of less smoothing on the score.

Alert: RuntimeRedactionActivityFlatlined

redacted_addresses_total is 0 for 30 minutes while ingest is active.

Top-3 likely causes:

1. Upstream agent symbol-format change — addresses now arrive in a shape the regex (@0x[hex]+) no longer matches. Verify the agent build and update RuntimeSymbolRedactor.AddressSuffix if needed.
2. All tenants disabled the agent simultaneously — coincidental but possible during a coordinated rollback. Cross-check traces_ingest_total{status="accepted"} by tenant.
3. Redactor regex performance regression — the regex has a 50ms timeout; if it’s hitting it, RuntimeSymbolRedactor.Apply may be silently returning the input. Capture a sample frame from logs and run it through the redactor in isolation.

Alert: RuntimeRetentionWorkerStalled

retention_deleted_total is 0 for 2 days while ingest is active.

Top-3 likely causes:

1. Worker disabled in config (findings:ledger:runtime:retention:Enabled = false).
2. Tenant discovery or deletion failed — with an empty Tenants list, the worker discovers stored tenant identifiers, including expired-only tenants. A nonempty explicit list limits selection; PerTenant keys are included in either mode. Check database connectivity and the worker logs.
3. All trace data already within retention window — possible if the feature is newly deployed or all traces are <90 days old. Verify by inspecting the oldest captured_at for the tenant with a read-only query.

Investigation:

1. Check service logs for eventId 1110 (Runtime trace retention swept) and 1111 (Runtime trace retention sweep failed).
2. Inspect configuration: findings:ledger:runtime:retention:*. An empty Tenants list enables stored-tenant discovery. The default retention and sweep interval must be positive; the initial delay cannot be negative.
3. If the worker has crashed at startup, the BackgroundService host’s crash log will explain why; restart the service.

5. Common operational scenarios

Scenario: Spike in traces_redacted_addresses_total

A rise (the opposite of RuntimeRedactionActivityFlatlined) means the upstream agent is suddenly producing more address-suffixed symbols than before. Common causes:

• Symbol resolution regression in the agent (failed to demangle).
• Changed sysroot in a base image — debuginfo missing.
• A specific tenant rolled out a new agent build.

Action: identify the tenant via topk(5, sum by (tenant) (rate(...))) and ping the tenant owner with the trace-symbol diff.

Scenario: Spike in traces_ingest_total{status="rejected_validation"}

Find the offending tenant: topk(5, sum by (tenant) (rate(findings_ledger_runtime_traces_ingest_total{status="rejected_validation"}[5m]))).

The validation log line (eventId 1105 with reason=validation) carries the field that failed. 99% of the time it is frames empty, artifactDigest missing, or capturedAt zero.

Scenario: Tenant requests a retention override

1. Add a key to findings:ledger:runtime:retention:PerTenant: {"tenant-acme": "30.00:00:00"} (TimeSpan format, days.hh:mm:ss).
2. The override key already includes that tenant in the sweep; a duplicate entry in Tenants is unnecessary.
3. Reload service config (or restart). The worker reads IOptionsMonitor so a config reload picks up the change at the next sweep.

Scenario: GDPR data-access request

Per design, the runtime instrumentation pipeline:

• Persists: tenant id, finding id, artifact digest, component PURL, container id, redacted symbol/file/line, captured timestamp, aggregate entry-point evidence.
• Redacts: trailing @0xADDR address tokens from symbol and file.
• Does NOT persist: process command line, environment variables, request bodies, source code, register state, memory contents, request metadata, or container name.

For a tenant-scoped data-access request:

SELECT * FROM findings.runtime_traces WHERE tenant_id = $1;
SELECT * FROM findings.runtime_trace_aggregates WHERE tenant_id = $1;
SELECT * FROM findings.runtime_timeline_events WHERE tenant_id = $1;

Raw trace retention is not a complete tenant erasure procedure. Aggregates, timeline events, scores and ledger evidence remain retained independently. Use the separately authorized tenant lifecycle and data-disposition procedure for a deletion request; do not infer permission to delete those records from a trace-retention setting. See Findings retention and custody.

6. Configuration reference

{
  "findings": {
    "ledger": {
      "runtime": {
        "enabled": true,                  // sprint 011 — disable for clean test runs
        "retention": {
          "Enabled": true,
          "SweepInterval": "1.00:00:00",  // 24h
          "StartDelay":    "00:05:00",    // 5m
          "DefaultRetention": "90.00:00:00",  // 90 days
          "PerTenant": {
            "tenant-acme":   "30.00:00:00"
          },
          "Tenants": []                 // discover all stored trace tenants
        }
      }
    }
  }
}

6.1 Migration 013 — best-effort has_entry_point backfill

Sprint reference: SPRINT_20260429_023_Findings_runtime_contract_corrections (column add) + SPRINT_20260429_024_FindingsLedger_runtime_hygiene task FL-RUNTIME-HYG-002 (capacity / caveat documentation).

Migration 013 (013_runtime_aggregates_entry_point.sql) adds the runtime_trace_aggregates.has_entry_point column and, in the same transaction, attempts to backfill the flag on existing rows by joining back to runtime_traces and looking for a frame that is BOTH the entry point AND the vulnerable function for the aggregate’s symbol.

Capacity note. On large runtime_traces tables the backfill is dominated by the JSONB EXISTS subqueries — empirically ≈ 1 second per 10,000 matching aggregate rows on a warm cache. Run a row-count estimate before applying the migration:

-- pre-migration: count aggregates that may be backfilled
SELECT count(*)
  FROM findings.runtime_trace_aggregates
 WHERE has_entry_point = FALSE;

-- pre-migration: count source traces available to drive the backfill
SELECT count(*) FROM findings.runtime_traces;

EXPLAIN for the backfill query typically shows the planner using the (tenant_id, finding_id, artifact_digest)-style index on runtime_traces for the inner join, with a function scan over each trace’s frames JSONB inside the existence check:

Update on runtime_trace_aggregates agg
  ->  Hash Semi Join
        Hash Cond: (agg.tenant_id = trace.tenant_id
                AND agg.finding_id = trace.finding_id
                AND agg.artifact_digest = trace.artifact_digest)
        Filter: ((SubPlan: jsonb_array_elements(trace.frames) frame
                  WHERE frame @> '{"isEntryPoint":true}')
             AND (SubPlan: jsonb_array_elements(trace.frames) frame
                  WHERE frame @> '{"isVulnerableFunction":true}'
                    AND frame ->> 'symbol' = agg.vulnerable_function))

The migration is idempotent: the has_entry_point = FALSE guard ensures re-running it touches zero rows after the first pass.

Best-effort caveat. Aggregates whose source runtime_traces rows have already been retention-pruned cannot be backfilled and remain FALSE. This is acceptable per the post-sprint-023 product decision — the runtime score formula’s entryPointPresence component (30% weight, see ADR-016 Findings Runtime Score Formula) will under-weight these aggregates until fresh traces arrive that re-establish the entry-point evidence. The condition is self-healing: the next ingest for the same (tenant, finding, artifact, vulnerable_function) that contains an entry-point frame flips the flag via the upsert path in PostgresRuntimeTraceRepository.UpsertAggregateAsync.

7. Related dashboards and rules

• Dashboard: devops/telemetry/grafana/findings-ledger/runtime-instrumentation.json
• Alert rules: devops/telemetry/grafana/findings-ledger/runtime-alerts.yaml
• Capacity bench: src/Findings/__Tests/StellaOps.Findings.Ledger.Application.Tests/RuntimeInstrumentationCapacityBench.cs
• Retention worker tests: src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/RuntimeTraceRetentionWorkerTests.cs
• PostgreSQL retention acceptance: src/Findings/__Tests/StellaOps.Findings.Ledger.Tests/RuntimeTraceRetentionIntegrationTests.cs

8. Change log