Vulnerabilities event storage

Vulnerabilities stores complete logical outbox envelopes in bounded gzip batches in its own database. The physical layout preserves every event ID, stream, epoch, sequence, type, version, occurrence timestamp and PostgreSQL-normalized payload string. Consumers continue to receive individual DC-06 envelopes from the existing event APIs. Compression never replaces the stream with summary events.

AddVulnerabilitiesPersistence selects this layout for both owner roles. Other services keep the neutral library’s existing row layout and migration chain. Inbox rows and event IDs, consumer checkpoints, remote registrations, leases and the stream allocator retain their current storage.

eventing.outbox_batch holds at most 1,024 consecutive envelopes per row. An approximate plaintext size target splits smaller chunks; a single larger envelope remains complete. Version and length metadata, gzip integrity, count and envelope identities are checked when decoding. An invalid batch fails the read; no event is skipped. Append allocates the same contiguous sequence range and commits the encoded chunks in the caller’s source transaction. The write path preserves the existing JSONB text representation and timestamp precision before encoding. Invalid JSON still aborts the whole source transaction.

Catch-up can start inside a batch and returns the requested logical suffix in sequence order. Publication splits a partial prefix without generating new event IDs. The return value remains the number of newly published logical events. Retention also splits exact sequence and pass-budget boundaries. The existing local/remote consumed floor, publication guard, epoch rules and retention windows still apply to each logical event. Cutoff comparisons retain PostgreSQL’s microsecond precision. The pass clock is checked between physical mutations; a partially completed pass reports its actual logical deletes and the exhausted time allowance. Retention reports and capacity event rates count events; physical chunk count is a separate storage fact. The owner selector fence reads the batch horizon inside the same repeatable-read snapshot as its page.

The owner adds forward-only migration 029_lossless_outbox_batches.sql. The immutable shared eventing chain remains unchanged. Startup runs the owner migration, then the shared eventing migrations and the awaited compressed outbox conversion, before provenance and product conversion. The narrow AddEventingStartupConversion seam registers only that schema/conversion pair; the later full reliability registration reuses it and leaves primitives and retention at their existing positions. No pruning cadence advances with the outbox conversion. Conversion uses an exclusive session fence and renames the old table, so a binary that still expects row storage fails closed. Each bounded transaction inserts complete encoded envelopes and removes the exact source rows together. A failed startup resumes from the remaining conversion rows. Startup drops that temporary source before serving. No compatibility view, dual writes or parallel rollback table remains after convergence. Both owner roles must use the new source in an upgrade window.

Earlier source retirement is an ordering choice, not a measured saving or a transient-space bound. Compressed target/index bytes, WAL and temporary allocation still need actual admission under the unchanged capacity limits. Existing images keep their embedded order until separately rebuilt and admitted; this source contract does not establish a shared-estate deployment.

The in-host event-rate probe and check-data-duplication.py use the same logical ledger identity and cutoff for both layouts. They reject an incomplete conversion instead of measuring only its partially populated target. The default row layout and absent-ledger signal remain available.

An image rebuild is required to carry the new embedded SQL and conversion code. A restart of an older image cannot perform this cutover. Shared-estate deployment remains a separate operator action.

Compression does not establish the customer footprint target. SIF-2 measures a full pinned seed after complete import, projection and publication, with the retained event burst, inbox, custody, all indexes and TOAST included. Seed and retained corpus files and WAL are reported separately.

The 2026-10-23 initial full-seed comparison passed all 80 gates with 24,935,978,687 candidate database bytes and M9 at 4.817880213750253. This includes complete retained logical events under the unchanged comparison contract. The complete delta/replay/custody and post-delta proof also passed, preserving all 21,085,337 original event envelopes and the two admitted fact events. Fresh-machine installation and ordinary configured OSV refresh were not exercised by this isolated measurement.

Storage source verified on 2026-10-20 at 5bfa7d015a183ebb782c1cc34a30e4a3cca2f149; the source test receipt records exact scope, executed cases and remaining acceptance.

Recheck the neutral CompressedOutboxTests, owner CompressedOutboxOwnerTests and IssueSelectorPagingTests.CompressedOwnerContinuation_UsesTheBatchHorizonInItsPageSnapshot with tools/scripts/test-targeted-xunit.ps1 and -BuildProjectReferences. Source tests establish storage and API semantics; the paired full-seed measurement establishes M9.