Raw-document lifecycle and storage measurements

The hub keeps exact source bytes for normalization, provenance and retained replay. Provider raw documents and their superseded copies are not customer mirror-seed contents. The customer imports advisory claims and selected metadata through the same owner ingestion path.

Verified against source commit 0fcdce5cc7ec52154230886ddb15fc894f97bb70, with the VRD changes tracked in docs-archive/implplan/SPRINT_20261007_061_Vulnerabilities_raw_document_revision_dedup.md. The source paths and re-verification commands below distinguish implementation from measured estate state. The six-source HSR-8 measurement is qualified against fa3e5dc12cbd7e4f164a9d431bc2c082c5316727; its 2026-10-13 measurement and stated limits were independently accepted.

Historical VRD qualification: the source behavior passed 27 focused safety checks, the full 199-case ingestion suite, 25 persistence cases and 11 adjacent producer cases. The approved native master sweep is complete. VRD-4 recorded the completed, dated from-empty customer footprint, and SIF-1 adopted it as its baseline. Standing Worker deployment and maintenance history is recorded separately under PCD-337; source checks do not establish live adoption. The reviewed raw declaration passes sprint 062’s identity checker. The sweep freed space inside the relation without shrinking the relation file. Sprint 061 VRD-6 rewrote the relation on 2026-10-09 (see below) and set the raw budget from that measurement; the raw budget pin is retired. OSV content identity was added by sprint 063 HSR-5, extended to the order of affected entries by HSR-9, and is verified at source level by its fixture-pair and ingestion tests.

Mirror delivery, current-record custody, exact provenance storage and complete outbox storage are source-verified against 727dcc45fb9faa361fb6480f082855fcd5fe3b58 (2026-10-23). The mirror-seed source anchors and re-verification commands identify that review’s scope. The 2026-10-23 initial paired proof establishes full-seed equivalence and initial candidate M9 at 4.817880213750253. The complete delta/post-delta proof also passed with that same whole-database M9. The historical VRD/HSR results below retain their original scope.

Content identity and normalization

The raw digest remains SHA-256 of the exact uncompressed document payload. CSAF content identity additionally excludes these regeneration fields:

• document.tracking.current_release_date;
• document.tracking.generator;
• document.tracking.revision_history[].date.

OSV (application/vnd.osv+json) content identity excludes the top-level modified and schema_version fields. A schema_version-only change is not a content change (owner ruling 2026-10-07). Nested fields with the same names, for example inside database_specific, remain content. Measured on the dev estate on 2026-10-07, before the VRD-2 sweep, 24% of stored OSV copies differed from their predecessor only in these two fields (4% modified, 20% schema_version).

OSV content identity also orders the top-level affected entries by their canonical JSON form before hashing; the order of that array is not content (owner ruling 2026-10-09). Duplicate entries are kept. The order inside an entry (ranges, events, versions, ecosystem_specific) remains content, and an affected value that is not an array is hashed as it is. Measured read-only on the dev estate on 2026-10-09: 13,914 Ubuntu OSV (UBUNTU-*) rows arrived in 24 hours, 13,543 of them further copies of a stored record. In a sample of 200 of those consecutive pairs (100 of the 10,888 pairs of equal length and 100 of the 2,655 others), 100 and 2 respectively differed only in the order of affected (and in modified/schema_version), about 81% of the pairs. The other 98 were content changes: new fixed versions with their notice references, added or removed packages, binary lists, priorities, severities, published and details. No sampled pair differed only in the order inside an entry or in a database_specific value, and no other ecosystem showed the reordering (0 of 748 consecutive GHSA, MAL, DRUPAL, GO, RUSTSEC and PYSEC pairs whose lengths differ by 16 bytes or less).

Each format’s hash input starts with a framing string that names its canonicalization (csaf-content/v1, osv-content/v2), so hashes of different canonicalizations never compare equal. HSR-5 cached osv-content/v1 hashes. When a cached OSV hash differs from the sighting’s, the writer recomputes the latest copy’s hash from its stored payload before the sighting counts as a change, and stores the recomputed value. A stale hash therefore neither stores a reordered copy nor masks a content change; each identity’s first sighting after the framing change reads one payload.

The unsigned Red Hat and SUSE connector streams (redhat-csaf, suse-csaf) and the OSV connector (osv) opt into their format’s content identity through SourceDocument.IgnoreVolatileFields. The option defaults to false. Every other source keeps exact-byte identity: cve, nvd, kev, debian-security-tracker, alpine-secdb, distro-binary-corpus, mirror import (stellaops-mirror) and signed issuer submissions. Sprint 063 HSR-8 measures CVE, NVD, KEV, Debian tracker, Alpine secdb and mirror import. Signed submissions and distro binary-corpus inputs remain outside that six-source measurement. Signed issuer submissions retain exact payload addressing and their signed-body provenance. Changes elsewhere remain content changes, including revision descriptions, product trees, vulnerability claims and the content of OSV affected entries. Malformed or ambiguous JSON must not suppress ingestion as an unchanged advisory.

The HSR-8 measurement found no demonstrated retrieval-only regeneration within its retained populations and bounded samples: 64 CVE pairs, all 463 retained NVD pairs, all 1,690 KEV pairs, 64 Debian pairs and all eight Alpine pairs. The provider-pair comparison preserves arrays, duplicates and JSON types; it is not a blanket consumer-equivalence proof. CVE/NVD source timestamps feed published metadata, KEV revisions add content such as forensicTriage, and sampled distro revisions change release/package claims. No new canonicalizer was justified. For mirror, 32 complete advisory groups projected offline from two real completed Hub publications were byte-identical. Those are bounded source-snapshot projections, not two historical wire exports or live import checks. The receipt records the coherent snapshot, payload/selection limits and excluded groups. Missing or swept history does not establish an all-time absence of regeneration; the existing byte policy remains unchanged.

The writer compares content with the latest retained document for the same source and upstream reference. An unchanged observation preserves the retained payload and digest, records the source-generation sighting, and skips normalization. It therefore does not repoint fact provenance to a timestamp-only copy. A real content change follows ordinary raw storage and normalization. An opted-in connector’s reappearance of older exact bytes restores its current claims through a new normalization position. Other inputs preserve their existing exact-byte behavior; mirror imports retain their explicit replay path. Historical rows without a content hash, and OSV rows whose cached hash has an earlier framing, are compared lazily; no whole-store payload rewrite is required.

An explicit mirror-seed delivery has a separate application boundary. After catching up previously pending mirror work, the owner applies the delivered records in bundle order, atomically with each raw admission and generation account. This permits an authorized value restoration even when its bytes were retained earlier. The original observation times remain unchanged, and the checkpoint never retreats. Identical delivery still produces no changed fact revision or event. An interrupted delivery retains only its committed prefix and cannot authorize completed replacement or cleanup.

See Ingest/IngestStores.cs and Ingest/RawDocumentContentHash.cs under src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Ingestion/, and src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Facts/Store/NormalizationPump.cs.

Retention boundary

The owner approved this custody change on 2026-10-07. Forward migration 019 supersedes migration 013’s raw-document no-sweep declaration. Applied migrations 001 and 013 remain unchanged. The P13 class remains source-of-truth: retained inputs are authoritative, and the sweep introduces no age expiry. The class label alone does not establish all-format identity or capacity conformance. DIP-2 owns the raw declaration; DIP-4 owns global canonicalizer fixture coverage and DIP-5 owns capacity receipts and enforcement. The original VRD-1 source scope covers unsigned Red Hat/SUSE CSAF; sprint 063 HSR-5 adds OSV. The reviewed declaration records each source’s identity and the remaining measurement/implementation owners. The raw capacity pin was retired by VRD-6; that retirement does not establish unmeasured format equivalence.

Retain all of the following:

• the latest document for each source and upstream reference;
• documents whose normalization is incomplete;
• documents with unknown or absent upstream identity;
• every digest referenced by fact provenance, advisory metadata, exploit evidence or fix evidence.

Only normalized, superseded, unreferenced documents are eligible for bounded deletion. The sweep coordinates with input writers and corpus publication. Reference-table locks protect the custody probes. The restrictive advisory-metadata foreign key supplies an additional database barrier. No raw-document deletion retracts a fact or deletes a fact revision.

The worker runs retention independently of online connector scheduling, including mirror-only installations. Before taking source fences or reading a source, retention tries the publication store’s local admission without queuing. An active local publisher or retention pass defers the source immediately. An admitted pass still holds the same exclusive filesystem lease as publication for its whole walk; this keeps an in-progress publication snapshot and its resumable sections protected. Vulnerabilities:Ingestion:RawRetention defaults to enabled, 500 documents per batch, 10 batches per source pass, 2,000 documents per window and a 15-minute interval. Startup validates positive bounded batch, pass and window limits and an interval of at most one day. Disabling retention preserves accepted inputs.

A source pass walks the source’s normalized documents in normalization order, one window at a time. A window read takes no reference-table lock and examines at most the window size, so its cost does not grow with the source. Only a window that finds eligible copies locks the reference tables, re-checks those copies and deletes them, at most one batch per transaction. A copy that gained a reference after its window was read is kept. Each window read and each delete has a 60-second client budget, and any lock wait inside them ends after 5 seconds. A lock wait that reaches that bound, like busy ingestion, normalization, publication or custody writers, defers the source to a later pass. An empty pass issues no delete and does not rotate the corpus input witness.

Checkpoint replay covers only retained raw inputs. The current producer database keeps facts.fact_revision; a customer seed does not carry the producer’s revision history. The owner’s customer replay substrate is the digest-verified seed file set kept by the operator (sprint 065). Producer raw history and customer seed replay have separate custody boundaries. The current seed import restores mirrored claims and retains current delivery records under the existing custody rules. It neither introduces a new local revision-retention window nor guarantees arbitrary historical generation reconstruction.

Vacuum can make deleted copies’ allocated pages reusable. It does not promise immediate shrinkage of relation files, Docker storage or the host VHDX. A table rewrite or VHDX compaction is a separate operator action.

Dev-estate sweep, 2026-10-07

The native owner maintenance runner removed 735,047 normalized, superseded, unreferenced copies from the two approved sources, with 99,446,118,255 stored payload bytes disposed. Both sources reached non-deferred passes with no eligible copies left.

All 32 sampled referenced digest identities remained present; the metadata FK stayed validated. Latest/upstream counts, native checkpoints, publication identity and original service health/mounts passed their controls. The publication input epoch rotated as required, while retained publication 215 remained unchanged. Source/custody contention deferred safely, including waiting for PostgreSQL autovacuum to finish. The exact owned runtime staging directory was removed after its process exited.

Physical raw relation size was 114,922,913,792 bytes before and 115,361,898,496 after the new indexes. This is logical payload disposal; relation files and the host VHDX were not rewritten or compacted. See the executed retention receipt.

Dev-estate relation rewrite, 2026-10-09

An owner-authorized ops window rewrote ingest.raw_document with VACUUM (FULL, VERBOSE, ANALYZE) after stopping the Vulnerabilities web and worker, the only writers. The rewrite held the table’s ACCESS EXCLUSIVE lock from 05:28:56Z to 05:47:08Z. VACUUM FULL runs as one transaction: a failure or cancellation leaves the old relation in place.

A fingerprint over every non-payload column was identical before and after. After the restart, SHA-256 of every stored payload equaled its digest for all 1,510,542 rows. The space returns to the Docker data disk’s file system, not to the host: the host VHDX does not shrink without a compaction. The raw budget is now 30,064,771,072 bytes and 3,100,000 rows: twice the measurement, rounded up. Until the standing Worker adopts content dedup, regeneration copies extend the rewritten relation by about 2.4 GB a day. See the rewrite receipt.

Compression decision, measured 2026-10-07

Retain PostgreSQL pglz TOAST and exact application payload bytes. Content deduplication and reference-safe retention address regeneration churn without changing every payload reader. The fixed sample below does not establish a corpus-wide compression ratio. Two large Red Hat documents contribute about 98% of its 24,335,083 bytes.

Lz4 increased stored bytes 7.76% while using less PostgreSQL CPU in this sample. Gzip Optimal saved 33.64% against pglz payload storage. Gzip would require a payload-format migration and explicit decompression at every direct reader. SmallestSize saved only 0.31% against Optimal while using 2.43 times its compression CPU. No application encoding or compression dependency is added by this decision.

TOAST used PostgreSQL 18.1 in an isolated, no-network, one-CPU container with tmpfs storage. The input table used uncompressed EXTERNAL storage. Each codec ran one warmup and seven measured inserts; pg_column_compression verified each selected codec. Read measurements repeated ten full detoasts through get_byte(payload, 0). Backend /proc user and system counters measured CPU at 100 Hz.

Gzip used .NET 10.0.11 on Windows, one warmup and seven process-CPU repetitions. Read measurements repeated ten full decode and array materializations. Every decoded payload passed SHA-256 round-trip verification. Windows CPU granularity and PowerShell overhead remain measurement limits. Linux PostgreSQL CPU and Windows .NET CPU cannot be directly compared.

The sample identities are fixed by exact uncompressed SHA-256:

The two RHSA-2026:4276 payloads are unchanged public source copies used as test fixtures. Their attribution and license remain beside the test fixture tree. The other measured payloads are not vendored or added to release artifacts.

Current customer baseline and approved target

The v3 mirror seed contains projected advisory claims and selected metadata from one pinned publication. It carries neither upstream raw-document history nor the producer’s fact revisions. Import stores each mirror record as an owner raw document and derives current facts, provenance, local revisions, consensus linksets and events. The VRD-4 baseline source also stored distribution statements; sprint 063 HSR-1 has since retired that table, and the VEX route computes them on read. The worker then publishes a retained corpus for local consumers.

These layers are the path measured by VRD-4 and adopted as SIF-1’s dated baseline. Current readers use them for metadata provenance, key convergence, normalization and publication. The raw sweep also bounds unreferenced superseded mirror records after changed advisories arrive through a delta. The owner seed import reclaims obsolete records for its delivered advisories after the complete data/cursor application, using the same normalized/superseded/custody checks. Import cleanup protects the delivered continuation keep-set; independently referenced older records remain. The general sweep still rechecks its existing predicate rather than retaining every unreferenced payload forever.

A mirror record’s current mirrorSeedStateDigest metadata binds its claims to the complete advisory-index state that delivered them. Identical bytes can receive a new delivery binding without changing their exact raw digest. Older independently held deliveries remain custody, not authority to restore retired claims, origins or metadata through maintenance. Explicit delivery can restore values. Metadata candidates still require their exact document digest and live fact backing.

The owner stores each exact origin pair once and retains every fact association and its proof/time. An ordinary read view reconstructs the original provenance fields. Complete outbox envelopes use owner-only gzip batches; logical identities, inbox/checkpoint storage and retention windows remain unchanged. See provenance storage, event storage and their upgrade prerequisites.

The owner approved a different target in SPRINT_20261007_065_Vulnerabilities_customer_seed_footprint.md. SIF-1 adopted the completed VRD-4 baseline and set the database/seed byte ratio target to at most 5.0. SIF-2 selects current-record supersession with continuation and independent-custody preservation. Digest-verified seed files remain the operator’s current-state replay input, not a historical database backup. Local revision policy is unchanged. This storage path does not claim a raw-free import or an absolute one-record limit per advisory. Sprint 063 HSR-1, which retired the stored distribution layer, is complete. Sprint 061 measured the pre-HSR-1 path. It does not establish the newer storage path’s footprint. The 2026-10-23 initial paired measurement passed with 24,935,978,687 candidate database bytes against 42,498,266,815 control bytes from the same complete 5,175,715,788-byte seed. The unchanged candidate maximum is 25,878,578,940 bytes. The delta/replay/custody and post-delta proof passed; the complete post-delta database remains 24,935,978,687 bytes. This isolated run does not qualify fresh-machine installation or an ordinary configured OSV refresh job.

Customer sizing requires a from-empty seed import, completed projections and final per-table pg_total_relation_size measurements. Producer raw storage and transfer size alone do not establish the customer database footprint. See mirror seeds, hub sizing, and the sprint’s isolated import receipt.

The VRD-4 full seed export completed with 387,212 advisories and 12,400,218 admitted claims. Its body is 4,905,152,865 bytes, with SHA-256 976fa4dc869249f8b49f18e4162bb238e27ae48477acb60ddc3da0c3c5684eba. The export pins source generation 213 and publication corpus-publication:174ca304d86bc5a080195dcbb0a6b9773318cb35978de5808e340fde92d9fb03. This measures the transfer input. The isolated from-empty import completed raw admission and normalization (387,214 records, 12,400,218 facts). After the consensus projection and owner publication completed, the run’s finalizer measured 44,908,713,663 owner-database bytes on 2026-10-08, a database/seed ratio of 9.1554157228. Of these, ingest.raw_document held 1,938,333,696 bytes (VRD-4 per-table receipt). An earlier interim measurement, taken at 62.45% of the projection (38,038,894,271 bytes), is superseded. The SIF-1 baseline receipt adopted the completed measurement and set a 24,525,764,325-byte limit for that original seed. That body and index are no longer available; its original publication’s two required sections have been reclaimed. On 2026-10-20 the owner approved a new full seed from the same source and OSV allowlists, pinned by digest, with paired unchanged/modified importer measurements. SIF-2 retains M9 at most 5.0: the new database limit is five times the new full seed’s bytes. SIF-1 remains historical evidence; no smaller sample or coverage reduction is permitted. Neither the old baseline nor the replacement-input approval proves the seed-native footprint target.

Re-verification

Run CsafRevisionContentHashTests, CsafRevisionIngestTests, OsvRevisionContentHashTests, OsvRevisionIngestTests and RawDocumentRetentionTests with tools/scripts/test-targeted-xunit.ps1 against the Vulnerabilities Ingestion test project.

pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Ingestion.Tests/StellaOps.Vulnerabilities.Ingestion.Tests.csproj -Class "*CsafRevision*"
pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Ingestion.Tests/StellaOps.Vulnerabilities.Ingestion.Tests.csproj -Class "*OsvRevision*"
pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Ingestion.Tests/StellaOps.Vulnerabilities.Ingestion.Tests.csproj -Class "*RawDocumentRetention*"

The real timestamp-only fixture pair must insert one row and leave provenance unchanged on its second observation. The real OSV modified/schema_version-only pairs and the constructed affected-reorder copy must do the same through the OSV connector, including against a stored row that has no cached content hash or a cached osv-content/v1 hash; the real OSV affected change must normalize against either. A changed unsigned advisory and an A-to-B-to-A restoration must still normalize. The signed-CSAF default-policy control must preserve exact payload digests. Retention tests must preserve latest, unnormalized, unknown-identity and referenced copies.

For seeded delivery, also re-verify MirrorRoundTripTests, ProvenanceOriginStorageTests and the complete outbox controls linked from mirror-seed source verification. Those source checks do not replace the full customer measurement or a deployment receipt.

Run tools/scripts/validate/check-p13-retention.py in default, strict, tables and owner modes, plus its self-test. Those gates verify source inventories; isolated safety tests and estate sweep receipts establish executed behavior.