ADR — Findings Runtime Score Recompute Async-Migration Trigger

Formerly docs/architecture/ADR-FINDINGS-RUNTIME-SCORE-RECOMPUTE-ASYNC-MIGRATION.md.

In one line: runtime score recompute runs in-line on the ingest hot path today, which is fine at current load; this ADR pre-commits the migration the on-call will execute when the RuntimeScoreRecomputeP95Threshold alert fires — a single-process Channel + BackgroundService, escalating to a durable outbox + worker only if multi-instance or crash-durability evidence demands it.

Audience: Findings Ledger engineers and the on-call responder for the runtime score-recompute alert. This is a deferred-decision ADR — it records the plan so nobody has to design under incident pressure.

Context

Runtime trace ingest currently runs the score recompute in-line: each POST /runtime/traces call writes the trace, upserts the aggregate, appends a timeline event, and then re-reads the recent aggregate set, runs RuntimeScoreCalculator, and writes the resulting score row before returning 202 Accepted to the agent.

This is the simplest possible design and currently meets the latency target (P95 ingest < 250 ms with Postgres in the loop; per the capacity bench in RuntimeInstrumentationCapacityBench, the in-process recompute itself contributes ≤ 20 ms P95 on synthetic load).

Observability captures the case where this assumption breaks:

• Metric: findings_ledger_runtime_score_recompute_duration_seconds{tenant} — histogram with normal P95 < 20 ms.
• Alert: RuntimeScoreRecomputeP95Threshold — fires when the P95 stays above 100 ms for 15 minutes (see docs/modules/findings-ledger/runtime-instrumentation-operations.md §4 “Alerts and response”).

Sprint 013 explicitly deferred the async-recompute work and sprint 014 shipped the alert. This ADR records the migration plan we will execute when the alert fires, so the on-call responder does not have to design under pressure.

Decision

When RuntimeScoreRecomputeP95Threshold fires sustained per the trigger conditions below, the default migration path is:

1. Channel + BackgroundService (single-process, in-memory).
   • Add a bounded Channel<(TenantId, FindingId)> to RuntimeInstrumentationService.
   • The ingest hot path enqueues (tenantId, findingId) and returns 202 Accepted immediately. Score-write moves out of the request.
   • A BackgroundService drains the channel, runs the calculator, and writes findings.runtime_scores. Deduplicates near-identical (tenant, finding) keys arriving in the same batch window.

The migration upgrades to outbox + dedicated worker (durable, horizontally scalable) only if a second observed problem demands it:

2a. Multi-instance Findings.Ledger deployments — multiple processes enqueueing into independent in-process channels would lose coalescing and produce redundant recomputes.

2b. Durability requirement after process crash — a crash between “enqueue” and “drain” loses the recompute pending for that batch until the next ingest for the same finding arrives.

We do not pre-emptively pay the operational cost (extra runtime_score_recompute outbox table, extra worker process, delivery-guarantee logic) of step 2 unless the data justifies it.

Trigger conditions (must be ALL true)

The on-call responder applies the Channel migration only when:

1. RuntimeScoreRecomputeP95Threshold fires, AND
2. P95 stays > 100 ms over a contiguous 1-hour window (the alert’s 15m window is the early indicator; a 1h window confirms it is not a transient), AND
3. No obvious external cause is the proximate driver:
   • Postgres incident (check pg health, replication lag).
   • Recent deploy event (correlate with deploy log).
   • Retention worker actively running large sweeps (check findings_ledger_runtime_retention_deleted_total{rate}).

If a proximate external cause exists, mitigate it first; the recompute itself is healthy at this scale.

Rationale

Why Channel + BackgroundService as the default

• Smallest possible change. ~1 day of work; no schema changes; no new external infrastructure; no new failure mode the on-call has not already encountered (Channels are idiomatic .NET).
• Removes recompute from the hot path entirely. The agent call returns 202 Accepted after the trace write + aggregate upsert + a channel send, all of which are sub-ms.
• Naturally coalesces. A burst of ingests for the same (tenant, finding) collapses to one recompute via dedup-by-key in the drain loop.
• Fits the existing wire contract. The score response (GET /runtime/score) already permits 404 NotFound for “no score computed yet”; clients already handle the eventual-consistency case cleanly.

Why not jump straight to outbox

• Adds a new table (findings.runtime_score_recompute) and a separate worker process — both of which are ops surface area we’d rather not pay for unless we have proof we need it.
• Two additional failure modes (worker crash, outbox-table saturation) that the on-call must learn.
• Complicates testing — every test that exercises ingest now must coordinate with the outbox.
• Loses simple observability — the in-process channel is a single metric (channel_depth); an outbox is a query against the table.

We expect the single-process Channel design to be sufficient for multi-year operation at our current per-tenant ingest volumes.

Consequences

After the Channel migration lands

• Score is eventually consistent. Stale by 1–30 seconds typically; bounded by the channel drain rate. Document this on the score endpoint (runtime-instrumentation-api.md §3) at the time the migration ships.
• In-flight queue at the moment of process restart is lost — the next ingest for affected (tenant, finding) keys will repair the score on its own, but a tenant with no traffic for a finding could see a stale score until they post the next observation. This is an acceptable trade for the simplicity gain.
• Hot path latency drops measurably. P95 ingest should fall from ~250 ms to ~50–100 ms (validated by the existing observability histogram findings_ledger_runtime_traces_ingest_duration_seconds).

Additionally if outbox migration is later needed

• Schema migration for the runtime_score_recompute table.
• Backfill for in-flight items at the moment of cutover — the Channel queue is in-memory, so cutover must drain the channel before switching enqueue paths. Ship this as a maintenance-window release.
• Worker process needs its own deployment, configuration, observability, and on-call runbook.

Alternatives considered & rejected

• Per-tenant rate-limited (debounced) recompute, in-line. Treats the symptom not the cause: under sustained ingest from one tenant, the debounce window introduces inconsistency that’s hard to explain (“I posted a trace 5 seconds ago; why is the score from before that trace?”). Rejected — complexity not justified at the expected load.
• Streaming / incremental score computation. The score formula’s recencyWeight and entryPointPresence components are aggregate functions over a time window; they do not lend themselves to clean incremental updates. Adding a “delta” path duplicates the calculator logic with subtle drift risk. Rejected — correctness risk too high.
• Skip the migration entirely; just scale Postgres. Score recompute is an in-process CPU concern (running the calculator), not a database-throughput one. Vertical-scaling Postgres would not move the needle. Rejected — addresses the wrong layer.
• Move recompute out of band via a periodic job (e.g. 1-minute sweeper). Loses the per-finding latency promise; clients reading the score immediately after ingest would see stale data for up to the sweep interval. Rejected — too much consistency loss for too little simplicity gain over the Channel design.

Test coverage (post-migration)

When the Channel migration lands, the following tests must accompany it (these are the binding asserts; the implementer adds them in the migration sprint, not now):

• RuntimeScoreRecompute_Channel_DrainsBatchedKeys — N concurrent ingests for the same (tenant, finding) produce exactly 1 recompute call.
• RuntimeScoreRecompute_Channel_EnqueuesAndReturns202BeforeRecompute — ingest returns 202 Accepted while the channel still has a pending item.
• RuntimeScoreRecompute_Channel_DrainEventuallyWritesScore — score appears within bounded time after enqueue.
• RuntimeScoreRecompute_Channel_ProcessRestart_LosesInFlightItems — documents the consequence of in-memory queue loss; the next ingest repairs the score.

References

• ADR-016 Findings Runtime Score Formula
• ADR-014 Findings Runtime Persistence
• ADR-013 Findings Runtime Disabled Mode
• docs/modules/findings-ledger/runtime-instrumentation-operations.md §4 — alert definition + initial action sketch.
• SPRINT_20260429_013_FindingsLedger_runtime_score — formula sprint that explicitly deferred this work.
• SPRINT_20260429_014_FindingsLedger_runtime_observability — sprint that shipped the trigger metric + alert.
• SPRINT_20260429_015_FindingsLedger_runtime_documentation Decisions & Risks — captures the deferral rationale.

Sign-off Window

Accepted on 2026-04-29 by Architecture lead. The 14-day default-accept window from sprint 024 (deadline 2026-05-13) closed early via explicit approval rather than the timeout path. Original window text retained below for history.

Following the project’s RFC default-accept process: if this ADR remains Proposed with no objection logged in this document by 2026-05-13 (14 days from sprint 024 commit), it is automatically promoted to Accepted. Architecture-lead may flip status earlier on review. Objections should be added under “Open Issues” below with name + date.

Open Issues

No objections logged during the sign-off window.