Vulnerability hub retention and custody

The hub retains source facts and accepted inputs separately from projections and bounded publication/replay state. RFC-6 records ordinary-table custody in forward migration 013_hub_retention_classes.sql. VRD-2’s forward migration 019 supersedes that migration’s raw-document no-sweep rule without changing its applied bytes. Neither change alters the facts.fact_revision partition migration, classification or retention behavior. Migration 023 (HSR-10) adds one disposal rule to that history: the revisions of a fact that no longer exists are pruned (Revision history).

The source declarations and extracted, hash-bound proof literals are in the retention source inventory. They do not claim that an unused schema declaration supplies a working product feature.

Retained records

Retained raw source bytes, current facts, provenance and connector state remain authoritative. Only complete source-generation reconciliation authorizes absence tombstones; age alone does not retract a vulnerability fact. A fact that a key-convergence rule tombstoned is a superseded identity, not an aged fact; it is reclaimed under the rule in Superseded identities. Normalization and metadata-backfill cursors preserve completed work until an explicit coordinated replay/reset.

Raw-document custody retains latest per source/upstream reference, incomplete normalization, unknown upstream identities and every digest referenced by facts, metadata, exploit or fix evidence. Only normalized, superseded, unreferenced copies may be swept in bounded batches. The class remains source-of-truth; no age-expiry window is introduced. Checkpoint replay covers retained raw inputs only. Fact revisions remain retained while their fact exists. See raw-document lifecycle for the owner ruling, writer behavior and storage measurements.

Binary/package attribution, captured fingerprints, fix evidence and symbol catalog records are retained inputs or outputs. An upstream URL does not guarantee that the same binary, normalization recipe or historical observation remains reconstructible. Build provenance parents remain because their output records reference them. The existing schema/diagnostic surfaces do not establish a completed build executor or corpus-curation writer where none exists.

The active corpus publication pointer is authoritative selection state. Its restrictive parent foreign key prevents deleting the currently selected publication. Existing bounded publication cleanup remains responsible for older publications.

Rebuildable projections and bounded state

The binary fact index derives its entire relation from canonical binary applicability. Consensus issue summaries are rebuilt by the fenced projector from live facts and provenance. VEX distribution statements are not stored: the VEX route computes them from live facts with provenance on read (see the retired derived table below). Advisory metadata candidates can be recovered from retained raw documents through the existing backfill path. Referenced raw documents remain protected; a removed historical input cannot be replayed after disposal.

facts.epss_change is a mutable, bounded replay cache. Every successful snapshot application prunes entries whose model date is older than that snapshot’s configured window. Vulnerabilities:Ingestion:Sources:Epss:ChangeWindowDays defaults to 30 and must be positive. Startup validation and the store reject zero/negative windows; invalid input cannot silently disable pruning. The SQL parameter is explicitly named retention_cutoff, exposing the existing model-date predicate to the P13 detector without widening the detector. This is a model-date bound, not wall-clock expiry during an ingestion outage. Current EPSS scores and import receipts remain retained.

Corpus section descriptors follow the existing publication lifecycle: retained current/previous publications remain available, older publications become reclaimed tombstones, and expired tombstones are removed after the configured interval (14 days by default). Section expiry explicitly joins that parent window in the same transaction; the foreign key also cascades parent deletion. No independent section sweep can break a retained manifest.

Superseded identities

A fact’s id hashes its key, so FactKeyConvergence corrects a key by writing the fact under the corrected key and tombstoning the stale one once no provenance remains. Such a fact is a superseded identity (data identity standard D9). Since migration 023_fact_identity_map_and_revision_prune.sql (owner ruling 2026-10-09) the re-key is recorded once, in facts.fact_identity_map: one row per superseded fact and successor, written at the tombstone instant (mapped_at = tombstoned_at). A successor is a fact the re-normalized document asserts under an origin (kind and reference) that the superseded fact lost, narrowed to the closest identity: the same applicability under the superseded key plus scope qualifiers (SPRINT_20261007_063 HSR-16, the CVE and NVD scoped retained identity), else the same product key and vulnerability, else the same product key, else the same vulnerability. A superseded fact with no successor gets one row with a NULL successor. The re-key writes no tombstone revision for the superseded fact and no first revision for its successors; an inserted fact that no superseded fact maps to is not a re-key and keeps its first revision. The outbox events are unchanged: one vuln.fact.tombstoned per superseded fact and one vuln.fact.upserted per successor. Before migration 023 the tombstone was recorded by a facts.fact_revision row with the same timestamp and revision hash and a change reason beginning key-convergence:. Migrations 022 and 023 restate the facts.fact_current class: “no age sweep” stays, and identity supersession is not age.

SupersededIdentityReclaimer (hub worker, lease vuln.superseded-identity-reclaim) deletes a superseded identity, with any facts.fact_provenance, binaries.binary_fact and facts.fact_identity_map rows that reference it, when all of these hold:

The revisions of a reclaimed fact are pruned after it (Revision history); every other revision keeps its 24-month window. GET /facts/{factId}/evidence reads provenance only, and a key-convergence tombstone has none, so its answer is the same before and after reclamation. A later document that asserts the same identity again writes it as a new fact. The probe of the pre-023 tombstone revision is transitional: it is removed once the estate holds no tombstone recorded that way.

A pass runs at most MaxBatchesPerPass (200) transactions of BatchSize (1,000) tombstones, every Interval (30 minutes), on by default like the hub’s other retention paths; Vulnerabilities:SupersededIdentityReclaim:Enabled=false holds it back during an incident. Each batch locks its window with SKIP LOCKED, deletes the superseded rows and adds its counts (including identity_map_rows) and cursor to the pass receipt in runtime.superseded_identity_reclaim_receipt (class cache-bounded, kept 400 days) in the same transaction. A later pass marks a receipt left running as interrupted and resumes after its cursor. Each batch starts on a freshly renewed lease and is cut at three quarters of the lease TTL (LeaseTtl, at least 4 s): a cut batch rolls back and the pass closes bounded, so no batch outlives its lease. A batch also waits at most 5 s for a lock another writer holds (lock_timeout), then backs off the same way: the corpus-input trigger’s compaction claim can hold every facts writer behind one long transaction. The partial index idx_facts_fact_current_tombstoned serves the walk, and the two tombstone records are probed once per row (LEFT JOIN LATERAL ... LIMIT 1); as correlated EXISTS in the select list the revision probe was planned as one scan of the whole revision family per batch (about 43 s per 1,000 rows on the dev estate’s first passes, 2026-10-09). The three foreign keys into facts.fact_current have an index leading with their fact id column.

Measured read-only on the local hub at 2026-10-08 00:08 UTC: 10,594,621 of 35,863,251 fact rows are key-convergence tombstones (one further tombstone is a mirror replacement), carrying 3.73 GB of row data (about 6 GB of the table with tuple overhead and their share of the indexes) and no provenance rows. Under the rule above, 15 were eligible at that instant, because the outbox window reached back only to 2026-10-01 00:08 and almost every tombstone was recorded from 2026-10-01 onward. The rest become eligible as the window passes them, through 2026-10-14, while publications continue. Space returns to the host only after a table rewrite and volume compaction.

Re-verify with the targeted xUnit helper for SupersededIdentityReclaimerTests (StellaOps.Vulnerabilities.Facts.Tests) and SupersededIdentityReclamation_HasItsTombstoneIndex_AndEveryForeignKeyItDeletesThroughIsIndexed (StellaOps.Vulnerabilities.Persistence.Tests).

Revision history

facts.fact_revision (class append-audit) holds one row per value revision of a fact: the first revision of an inserted fact and one per changed value (FactWriter), and a mirror replacement’s tombstone (MirrorClaimSetWriter). It is partitioned by month on changed_at and keeps 24 months, reclaimed by dropping whole partitions (facts.prune_fact_revision_partitions). A revision is retained while its fact has a facts.fact_current row, live or tombstoned. No route, corpus publication, mirror seed, compact export or other service reads the table; the reclaimer’s probe of pre-023 tombstone revisions is its only reader.

SupersededRevisionPruner (hub worker, lease vuln.superseded-revision-prune) deletes the revisions of facts that no longer exist: the revisions of identities the reclaimer deleted, including the re-keying revisions written before migration 023. A walk visits the history in primary-key order, one bounded batch per transaction fenced by the lease token: it reads BatchSize (10,000) keys, probes facts.fact_current by primary key for their fact ids, deletes the keys whose fact has no row (re-checked at delete time, so a fact asserted again keeps its history), and adds its counts and cursor to the pass receipt in runtime.superseded_revision_prune_receipt (class cache-bounded, kept 400 days). Each batch starts on a renewed lease and is cut at three quarters of its TTL, like the reclaimer’s. A pass runs at most MaxBatchesPerPass (200) batches every Interval (30 minutes); a walk over a large history spans passes, and a pass that dies is marked interrupted and its walk resumes after its cursor. A new walk starts only when no walk has completed, or when a reclamation pass that could still commit when the previous walk began (reclaim_mark) has deleted facts; otherwise the pass is idle. The reclaimer is the only path that deletes a facts.fact_current row, so no other event creates orphans. Vulnerabilities:SupersededRevisionPrune:Enabled=false holds the prune back during an incident. The prune never rewrites a table: the space of deleted rows returns only in an operations-window rewrite.

Measured read-only on the local hub at 2026-10-09 (0.3% sample of 158,436 rows): facts.fact_revision_legacy_pre_2026_10 holds 7.90 GB / 22.87M rows, 42.9% of them first revisions of key-convergence-tombstoned facts; facts.fact_revision_2026_10 holds 13.70 GB / 30.61M rows, 46.2% key-convergence tombstone revisions and 10.3% first revisions of facts tombstoned again. Once those facts are reclaimed the prune deletes about 27.1M rows; about 26.4M remain, one first revision per live fact (one sampled row recorded a later change). The partitions were rewritten on 2026-10-19 (VACUUM FULL, local hub): the family measured 10,229,858,304 bytes / 34,549,327 rows, more than that expectation because the hub then held about 35.1M live facts, each with its first revision. The data identity register declares the family as an opt-in (append-audit) entry from that measurement: 14 GB (the revision share of the 70 GB facts window) and 47M rows; the change value repeats by design (382,892 distinct values; at most 4,016,344 rows share one).

Re-verify with the targeted xUnit helper for SupersededRevisionPrunerTests and the identity-map cases of FactKeyConvergenceTests (StellaOps.Vulnerabilities.Facts.Tests), and FactIdentityMap_KeysOneRowPerPair_TreatsANullSuccessorAsOne_AndTheRevisionPruneHasItsReceipt (StellaOps.Vulnerabilities.Persistence.Tests).

Retired empty declarations

The current ingestion and projection paths do not read or write these tables:

Normalization reads raw documents directly; consensus uses the shared eventing outbox/inbox. Alias/product normalization occurs in code rather than these unused stores. Their retirement does not remove those active paths.

Migration 012 locks all targets before checking them, refuses any nonempty table with SQLSTATE 55000, and drops the empty set without CASCADE. If a populated target is found, preserve and review those records before a separate disposition decision. The binary Doctor census follows the remaining schema. If that schema’s table count is incomplete, it reports the counts and an unhealthy result before querying missing row-census tables.

Retired derived table

dist.statement (class derived-rebuildable in migration 013) held one statement per live, provenance-backed fact: the same values facts.fact_current already holds, plus a hash of the fact id. SPRINT_20261007_063 HSR-1 moved the VEX route to compute those statements on read, removed the projector’s rebuild of the table and added migration 020_retire_dist_statement.sql, which drops it. Unlike migration 012, migration 020 drops a populated table: its rows are derived, and the facts and provenance they were derived from remain authoritative. In the source inventory, both dist.statement declarations (migrations 001 and 013) are inactive declarations with basis retired-in-chain, and the table inventory no longer lists the table.

Re-verify with the targeted xUnit helper for VexStatementOnReadContractTests (StellaOps.Vulnerabilities.Facts.Tests: served statements equal the retired projection and the bytes the table served) and HubSchemaConventionTests (StellaOps.Vulnerabilities.Persistence.Tests), then run python tools/scripts/validate/check-p13-retention.py --repo-root . with and without --tables.

Verification

The VRD raw-document lifecycle is verified against source commit 0fcdce5cc7ec52154230886ddb15fc894f97bb70. The focused safety run passed 27 cases, including reference custody, enabled publication-lease contention, source/reference writer fences and the empty-pass witness control. Re-verify with the targeted CsafRevision*, OsvRevision* and RawDocumentRetention* classes in the Vulnerabilities Ingestion test project, using the lifecycle commands. Master adoption and the approved sweep remain separate execution boundaries.

Historical RFC-6 verification used source commit 32fe97a95071eed6e0ace56c7024f15e2565615b. The verification receipt records executed checks, negative controls and the source census.

Use the repository’s targeted xUnit helper for HubSchemaConventionTests, EpssRelocationContractTests, NormalizationPumpTests, CorpusAdvisoryMetadataBackfillTests, ConsensusProjectionHostTests and BinaryPlaneDoctorCheckTests. The retirement tests reject populated targets individually and verify that no sibling was removed before the refusal.

The verified source commit and execution counts are recorded in the RFC-6 receipt in the Program sprint. Source and isolated database acceptance do not imply a production rollout.