StellaOps.Vulnerabilities — module dossier
Currency. Reconciled 2026-09-11 against
src/Vulnerabilities/, the completed VULN-B1 source retirement and the BIN-4 implementation. The hub has served production traffic since 2026-08-04. The oldvuln,vex,vexhub, andconcelierschemas are dropped, their services are stopped, and the VexHub/VexLens/Concelier source trees are retired. BIN-4 source validation does not imply activation of binary-reference derivation in an estate. Target design and the full build record live in vulnerability-data-plane-v2-design.md.
1. What this service is
The vulnerability knowledge hub: one microservice owning one database (stellaops_vuln) that holds raw + normalized vulnerability, VEX and distro knowledge for the whole platform. It replaces Concelier (incl. Feedser + Excititor) and VexHub, which shared a database with the rest of the platform and had grown to 91 GB, ~98% of which was this domain.
Two properties define it, and almost every design decision follows from one of them:
It is tenant-free. There are no tenant columns, no row-level security, and no per-tenant gate decisions anywhere in stellaops_vuln. What the world says about a vulnerability is not a per-customer fact. Everything tenant-scoped — a tenant’s own VEX decisions, source-trust overrides, gate verdicts — lives in the control plane (Policy, Findings), which is why policy.vuln_gate_current exists and vuln.issue_gate_decisions does not.
It is rebuildable from its sources. Every derived table can be dropped and reconstructed from ingest.raw_document + a re-ingest. That is not an aspiration: it is the documented disaster recovery path (T12), and it is why the v2 build was a fresh rebuild rather than a migration — the v1 corpus was already pre-flattened past the point where the original claims could be recovered.
2. Source layout
Installation posture for online ingestion
The worker reads installation posture from its own local catalog replica before each online HTTP attempt. This covers NVD, CVE, OSV, KEV, Red Hat, SUSE, Alpine, Debian, binary corpus indexes, EPSS and package payload retrieval. A declared seal strengthens the existing local egress policy; a declared unseal cannot weaken a local seal or its destination rules. Missing or unreadable replica state refuses the attempt. A witnessed absence uses local bootstrap configuration. Local mirror-file import is unaffected.
Each retry and redirect passes through the posture check after acquiring any throttle slot. The transport disables automatic redirects; the connector’s bounded GET/HEAD redirect handler rechecks each destination, refuses HTTPS downgrades and strips credentials when changing origin. Local policy refusal propagates once as AirGapEgressBlockedException; it is not a publisher HTTP response or a transient transport failure.
Enable the drain through Catalog:Replication:EnvironmentState with authenticated client-credentials mode and an explicit AuthTenant. The worker reuses its own database and requires an existing service identity granted catalog:replicate. When no Auth.Client identity is already composed, configure Vulnerabilities:Authority:Authority, ClientId, ClientSecret and an absolute TokenCacheDirectory; TokenEndpoint is optional. Provision the credential through runtime secret configuration and mount a durable private writable cache directory. There is no anonymous feed, automatic identity enrollment or implicit memory cache. A new worker with an unconverged replica refuses online ingestion, so feed authentication and convergence belong to the deployment preflight.
| Project | Role |
|---|---|
StellaOps.Vulnerabilities.Contracts | Closed contracts SDK (DC-25). Zero ProjectReferences by contract; purity is conformance-verified, not assumed. |
StellaOps.Vulnerabilities.Consensus | THE consensus evaluator (DC-38) — one pure function shared by the hub (global consensus) and Policy (tenant-effective consensus). Also a classified client SDK, for the same reason: it is pure. |
StellaOps.VulnMatch.Core | DC-30 artifact-consumer SDK. The one deterministic, offline-capable SBOM-to-advisory matcher used by online and offline consumers. The assembly/namespace contract is preserved; its closed graph reaches only neutral version comparison plus the pinned Semver package and contains no database provider, HTTP, DI, host, clock, or configuration binding. |
StellaOps.Vulnerabilities.RawModels / .Normalization / .Canonical | Ingestion substrate carried over from the v1 tree where the transfer was purely mechanical: raw document shapes, version-range normalization, canonical domain primitives. |
StellaOps.Vulnerabilities.Persistence | The fresh schema baseline + startup migration wiring. Auto-migrates the service’s OWN database; fails closed without STELLAOPS_POSTGRES_VULN_CONNECTION. |
StellaOps.Vulnerabilities.Ingestion | Connector SPI, the ingest pipeline, the capacity guard, and first-party issuer submission. |
StellaOps.Vulnerabilities.Facts | The fact store, projectors, EPSS, mirror import/export, the corpus artifact, and the serving queries. |
StellaOps.Vulnerabilities.BinaryArtifacts | Closed offline SDK for native-code normalization, signature authoring/matching and predicate formats. Shared by the CLI and hub; no reference admission, persistence or issuer workflow. |
StellaOps.Vulnerabilities.BinaryAnalysis / .Binaries | Digest-bound reference generation, bounded package analysis, exact binary subjects and owner persistence. |
StellaOps.Vulnerabilities.Fingerprints / .DeltaSig | Hub-owned feature generators, feature storage, evidence policy and Symbols contract adaptation. |
StellaOps.Vulnerabilities.WebService / .Worker | The two deployables (vulnerabilities-web, vulnerabilities-worker). |
The family solution (StellaOps.Vulnerabilities.slnx) deliberately lists only family-owned projects. dotnet slnx auto-add pulls the whole transitive closure — trim it, or the solution stops being evidence of anything.
3. Data model — seven schemas in one database
One owner means internal schemas are fine; they are organizational, not ownership boundaries.
ingest—raw_document(content-addressed originals),dto_document,source_state,quarantine. This is the layer everything else can be rebuilt from.facts—fact_current+fact_provenance+fact_revision(partitioned history),alias,product_identity, and the EPSS trio (epss_import/epss_current/epss_change).consensus—issue_linkset(global, narrow: counts, status/severity summaries, content hash) +rebuild_queue.dist—statement(the former VexHub statements, now a rebuildable view over facts) andmirror_state.runtime—source_generation,reconciliation_audit, plus the sharedeventingschema (outbox/inbox/checkpoints/leases) homed here by the P6 reliability core.binaries— the binary-granularity knowledge plane folded in by DC-33:binary_identity,binary_package_map, the fingerprint/delta-signature corpus, the patch-awarecve_fix_index, and reproducible-build provenance.binary_factis not a verdict store — it is the resolution index from a binary applicability scheme (buildid/fingerprint) ontofacts.fact_current, because binary applicability is a fact class like any other.symbols— the hub-adjacent symbol CAS catalog (source,catalog_entry,manifest,blob_catalog). Blob bytes are in the object store undervuln/symbols/, never in this database (P16).
The identity rules worth knowing before you touch anything
A fact’s identity includes its applicability, and a revision hash is a value, not an identity. Two statements about the same (vulnerability, product) with disjoint version ranges are two distinct facts and must never collapse into one — that collapse is what flattened the v1 corpus. When a vendor republishes the same statement with a changed status, the fact keeps its identity and gains a revision; the old revision stays in history.
Provenance is many-to-one, and only the last origin’s removal tombstones a fact. The same statement arriving via a connector claim and via a derived distribution is ONE fact with TWO provenance rows. Removing one origin leaves the fact alone. Removing the last one tombstones it and rebuilds its linkset in the same transaction — there is no retention sweep in v2, and adding one would reintroduce the deletion-by-schedule behaviour the v2 design removed.
An issue linkset’s subject identity is (vulnerabilityId, productKey); its content hash is an observation revision. This is the consensus.issue_linkset primary key, with no tenant dimension. The values are ordinal and case-sensitive — especially productKey, where mixed-case Maven group IDs are ordinary — and consumers must preserve the exact producer-returned case. contentHash is SHA-256 over the canonical projection including live fact revision hashes, so a value change keeps the subject and changes its observation revision. Scanner’s worked evidence contract applies that distinction in advisory-evidence-identity.md; it never derives a Guid. SCN-ID-1 added the zero-reference producer response DTO plus OpenAPI parity. SCN-ID-3 now maps the product-issues endpoint through that closed DTO and advertises its typed 200 response. Scanner’s disabled-by-default adapter consumes only this zero-reference producer contract, preserves exact case/content hash, and emits no evidence. SCN-ID-4’s canonical v2 codec is DONE. SCN-ID-5’s authoritative producer has emitted genuine v2, but live acceptance remains blocked until its all-products-available bootstrap and forcing/rollback/soak pass; SCN-ID-6 retirement follows. Product-key path routes recover the final segment from Kestrel’s raw request target and decode HTTP transport exactly once. This preserves canonical PURL escapes such as OCI-name %2F and digest %3A; applying UnescapeDataString again to the mixed bound route value changes the producer-owned identity and is forbidden.
4. Reconciliation protocol
Every projection in this service follows the same four rules, and they are not independently negotiable:
- One source transaction commits the fact revision/tombstone and its ordered outbox row. Absence-tombstones are authorized only by a completed generation — a connector that died halfway through must not be able to delete what it simply failed to fetch.
- One leased writer per stream, with a fencing token checked INSIDE the destination transaction. A second instance blocks on the lease; an expired holder’s commit is rejected by its stale token rather than by hoping it noticed in time.
- One destination transaction per event: inbox admit → rebuild exactly the affected rows → advance the checkpoint. A failure rolls back all three. A checkpoint that could advance past a failed derivation would silently drop the event forever.
- Replay is physically idempotent. Content-guarded upserts mean a replay writes zero hot-row versions — asserted directly by summing
xminbefore and after a full re-consumption.
Trap, learned the hard way (2026-08-02): the producer-epoch check must run before any empty-batch early return. A rebuilt producer restarts its sequence at 1, so a stale checkpoint of (old epoch, 7) filters seq > 7 against the new epoch and legitimately returns zero envelopes. An early return there leaves the consumer permanently frozen on a stream that is producing normally. An empty batch is not evidence that your checkpoint is still valid.
5. What the hub serves
Under /api/vulnerabilities/v1/ (gateway route group added by SPRINT_20260722_006):
- readiness — generation-based, never wall-clock, because a gate evaluating against an empty corpus must be able to tell that apart from a clean result (DC-09).
readyis true once at least one source has a completed generation that ingested a document; a completed generation that saw nothing never declares a corpus ready. Readiness deliberately does NOT require every source to be green — that would hold a populated corpus atrebuildingfor one permanently-failing source, which is what per-source health is for. Backed byGenerationBackedCorpusReadinessProvider; the skeleton’s placeholder that answeredready:falseunconditionally was replaced bySPRINT_20260804_002FCR-1 after it was measured serving{"ready":false,"sources":[]}against a corpus of 309k facts. - sources — per-source ingest generation status over
runtime.source_generation: the latest generation and the last completed one, with a derivedhealthword. This is the successor to the retired plane’s advisory-source catalog + feed-freshness projection, and the Console’s Advisory & VEX sources page and dashboard feed lens both read it. The vocabulary ishealthy | ingesting | empty | stalled | never-completed, computed once here so no consumer derives a competing one — the Console G3 review found a connectivity dimension and an ingestion dimension disagreeing 9× over the same estate. Two of the words exist for states a “last run succeeded” view hides:never-completed(a source that has never once succeeded) andempty(a run that closed CLEANLY having ingested nothing — the silent-green state that, under D4, is what may authorize absence-tombstones). It is READ-ONLY: source administration is config-driven and its return is an open decision (SPRINT_20260804_002FCR-2). - lookup — advisories, per-product issues, per-issue facts, fact evidence, VEX statements. The per-product
IssueSummaryis a global linkset observation: tuple, fact/status/severity summaries, and content hash. It is not anEffectiveConsensusor tenant decision, so a consumer must not invent issuer precedence or suppress a match from the count maps alone. - corpus export — deterministic NDJSON, no BOM. (A
StreamWriterwithEncoding.UTF8emits a BOM that makes the first NDJSON line unparseable and changes the digest of identical content.) The stream declares what it is —X-StellaOps-Corpus-Section: matcher-rowsandX-StellaOps-Corpus-Format-Version— so a consumer can apply the n/n-1 window instead of assuming; an undeclared stream is refused, because assuming the current version is the silent misparse the window exists to prevent. Its rows are the same bytes as the artifact’smatcher-rowssection: one query, one serializer, both sides (HubQueries.StreamCorpusAsync+CorpusMatcherRowSerializer). The SQL orders with an explicitCOLLATE "C"— without it the row order, and therefore the digest of any artifact built from it, would depend on the database server’s locale, so two hubs holding the same corpus would disagree about its identity. This is measurable, not hypothetical: on anen_US.utf8databaseORDER BY xsortspkg:maven/Apache/x, pkg:maven/apache/y, pkg:maven/BSD/z, whileCOLLATE "C"sortsApache, BSD, apache— and mixed-case product keys (Maven group ids) are ordinary. The order is strict: exactly one row perCorpusMatcherRowOrderkey (vulnerability, product, introduced, fixed, status). Several issuers routinely assert the same fact shape (measured 2026-08-26: 354,860 duplicate keys over 16.5M facts, 13,375 with differing severity), and a consumer that verifies the order refuses a section with a repeated key. The producer therefore collapses a key in SQL (DISTINCT ONover the ordered stream, bounded memory) and picks the value fields deterministically — producer semantics, not a wire change (ruled 2026-08-26): the highest-authority issuer first (the advisory-metadata selector’s tier ordercve<nvd<redhat-csaf/suse-csaf<osv<stellaops-mirror< others, so one publication never carries two severities for one issue), then the greater severity, then the remaining ties by their bytes. Per-source divergence is whatconsensus-inputscarries; it never becomes a second matcher row. - EPSS — the hub owns EPSS (P14). Scanner and risk read it from here instead of holding their own ~2 GB copy.
- issuer submission — DSSE-enveloped OpenVEX push from allowlisted internal issuers, guarded by the
vuln:submitmachine scope and a per-issuer client allowlist, failing closed on signature verification. The configured Stella OpsIAsymmetricSignatureVerifierperforms verification; the service does not instantiate raw RSA/ECDSA primitives or override a registered verifier. Null envelope members are rejected without storing a document or fact. Private signing keys remain with the producer’s signing service; the hub holds public verification material only. The signed document is fully normalized before the transaction writes anything; invalid binary subjects return422 issuer_document_invalidwithout partially ingesting mixed input. Raw-document metadata retains the public key, payload type and signatures needed to verify the retained payload. Identical replay preserves fact identity and repairs a missing binary lookup projection without duplicating outbox events. - event catch-up —
GET /events/{stream}, the DC-07 pull surface. This is what lets the control plane consume hub events without ever openingstellaops_vuln. The response carriesretentionHorizonSeq(the oldest offset the stream can still serve) and, when the caller’safterSeqis below it, aretentionblock naming how many envelopes were reclaimed and pointing at the retained bootstrap leg (GET /api/vulnerabilities/v1/corpus/publications/current). For a below-horizonvuln.corpusrequest the block also embeds the active publication descriptor, so the consumer can immediately resolve its opaque handle without guessing a generation. The guard is non-blocking by owner ruling: the page is still served. It exists because a pre-window page is ordered and gapless in itself and so looks perfectly healthy while silently missing everything below the horizon — the event log is not the system of record, so the answer to a reclaimed range is to re-derive current state. - cross-database consumer registration —
POST|GET|DELETE /events/{stream}/consumers[/{id}]. A consumer that commits in its own database can never write a checkpoint here, so it registers and reports its durable position instead; that is what lets the hub bound a stream it does not consume itself without ever risking the consumer’s backlog. Registering pins the stream (a registration that has reported nothing pins ALL of it); reporting advances what may be reclaimed; the lease that decides when a silent registration stops holding the floor belongs to the hub, not the caller. Ridesadvisory:read— the caller may already read every envelope, and registration only ever makes the hub keep more data, so a separate write scope would put the bound behind an entitlement existing consumers lack. SeeOutboxRetentionClass.
Canonical OpenAPI baseline: src/Api/StellaOps.Api.OpenApi/vulnerabilities/openapi.yaml. The retained-publication routes are source-complete and specified below, but their generated API catalogue synchronization is separately owned outside SPRINT 010’s cross-module allowance; release must close sprint risk F-R0821-X18-2 before advertising them through generated API artifacts.
Measured binary reference derivation
The binary-plane Doctor census expects 11 binaries tables and five symbols tables after startup migration 009_symbol_management.sql adds symbols.catalog_manifest. Missing or additional tables produce an unhealthy census with measured and expected counts; folded-plane marker checks also reject a different database that happens to use the same schema names.
Source verification: 0e5792ebea77d6fff0f58ca6ee04d204737360c3. The BIN-4 acceptance receipt records the executed suites, recheck commands and estate boundary.
The worker can compare package code with operator-admitted vulnerable/patched reference pairs. Each pair declares a reference ID, CVE, component, ABI, target symbols and exact payload URLs and digests. Native payloads remain transient. Both binaries must contain every requested symbol, share their measured architecture/format and produce at least one different normalized code signature. Relocation alone does not establish a patch discriminator.
Complete exact comparisons can emit affected or fixed. Missing symbols, unknown code, conflicting states or an unknown/mismatched configured distro ABI produce no statement. The worker records the reference digests, recipe, measured symbol hashes and binary digest in the signed evidence. It does not infer not_affected from absence. Per-package observation limits are logged and mean remaining binaries received no derivation.
Enable through Vulnerabilities:Binaries:References; the shipped example configuration has an empty Pairs array and is inactive. At most 16 pairs are admitted, with 1–256 named target symbols per pair. DistroAbis is explicit source configuration, not measured ABI. The default observation limit is 64 binaries per package. Signed analyzer admission uses the existing BinaryIndex:RuntimePlugins configuration; the worker has no compiled B2R2 implementation.
Publication configures the Signer and hub base URLs, actual worker image ProducerDigest, service AuthTenant and an absolute mounted ProofOfEntitlementFile. Existing Vulnerabilities:Authority credentials request signer:sign and vuln:submit. Signer must trust that exact digest under TrustedProducerDigests["vulnerabilities-worker"], and its configured KMS key must match the public key enrolled for the hub issuer. The worker requests signing through Signer, checks the returned predicate and binary subject, then submits the DSSE proof to the normal issuer endpoint. A signing/submission failure holds the corpus checkpoint for retry. No private key enters the worker or hub.
Exact OpenVEX subjects use urn:stellaops:binary:<scheme>:<escaped-variant>:<escaped-value>. Build-id variants are gnu-build-id, pe-cv and macho-uuid. Fingerprints use an empty variant and preserve the opaque ID’s case. These identities survive consensus and full mirror transport; version-range matcher-rows and the compact vulnerability matcher exclude them.
6. The corpus artifact
The exploit-evidence consumer contract distinguishes curated exploitation, exploit availability and probability in Findings, Policy and offline exports.
Consumers that need to match against the corpus (Policy’s gate path, the CLI’s compact export, offline estates) do not query the hub. They consume a generation-stamped, sectioned artifact: matcher-rows, consensus-inputs, exploit-evidence, reachability-sinks, and advisory-metadata. Each consumer declares the sections it uses and fetches only those, which caps per-host disk without a second artifact format.
Compact exports additionally require three optional publication sections: compact-metadata (v2 metadata rows scoped to one source record, without long prose), compact-osv-documents (v1 OSV-shaped identifiers, aliases, withdrawal status and symbol-bearing affected entries), and compact-kev-documents (v1 current raw CISA entries, including ransomware status). They use the same publication snapshot and retention limits. The five mandatory v2 descriptors remain unchanged for other readers. The additive sourceStatementId provenance property in consensus-inputs identifies an upstream record for compact license routing; existing consumers may ignore it, while the compact exporter refuses a publication that omits it. Fact identity and consensus semantics are unchanged.
- The identity of a generation is its manifest digest, and the manifest deliberately contains no timestamps — a wall-clock stamp would break the property that regenerating the same corpus-as-of-G reproduces the same digest, which is the entire replay contract.
- The four legacy section readers implement an n/n-1 v1/v2 window and refuse anything outside it.
advisory-metadatawas born at v2 and accepts v2 only; there is no invented v1 metadata shape. Air-gapped estates are the design case, so every incompatible semantic row change bumps that section format rather than being silently accepted.CorpusReaderchecks the named section, and both raw metadata parsing and streaming validation require the attached descriptor’s format version explicitly; no metadata call path silently assumes v2. - The worker is the sole publication writer. Under one fixed cross-process lease it takes one repeatable-read database snapshot, streams the required and optional detached parts into the filesystem CAS, flushes and verifies every digest/length, and only then installs the catalogue/current pointer together with the v2 outbox envelope in one owner-database transaction. Only
completedgenerations are publishable; the web role mounts the same root read-only. - Cadence: at most ONE publication per ingest CYCLE (owner ruling 2026-08-27, sprint 003 VULN-B8 (a)). A publication is a WHOLE-CORPUS snapshot — the section streams take no source filter, so
sourceKey/generationonly gate completeness and stamp identity — so publishing per completed source generation wrote one complete snapshot of the same corpus per completed source, all but the last superseded almost immediately.IngestScheduler.RunOnceAsyncnow records a candidate stamp per completed non-binary-corpus source and publishes once, after every enabled source (and the EPSS snapshot) has been attempted, so the snapshot carries the freshest corpus the cycle produced. The identity scheme is unchanged: the publication’s unique key is still(source_key, source_generation), and the cycle’s one publication is stamped with the last source, in configuredEnabledSourcesorder, that completed a publishable generation in that cycle — deterministic, because the cycle loop is strictly sequential over that configured order. A cycle in which no source completed publishes nothing. Publication is a cycle-level outcome, not a source outcome: it is reported under the reserved keycorpus-publication, so a failed publication is never charged to a source whose own ingest succeeded, and it does not cancel the cycle. - The artifact contract itself (sections, version window, manifest canonicalization, reader) lives in
StellaOps.Vulnerabilities.Contracts, not in the hub’s domain library, because producer and consumer must assemble and verify identity with the same code — two implementations of “content-addressed” is two answers to what a generation is. stella vuln-db export --from-hub <url>writes an artifact from the hub’s stream: the section bytes verbatim (re-serializing would drop any additive field a newer hub added and would make the CLI’s digest disagree with the producer’s) plus the canonical manifest. Such an artifact is an unstamped snapshot — its manifest carries nogeneration, because the hub has no single global generation number and claiming one it was never stamped with would break the replay contract that generation identity exists to serve. Snapshots therefore cannot enter a matcher host’s A/B working set, which is keyed on generation.
6.1 Advisory metadata and generation publication wire
The producer-owned closed SDK freezes the first X18 slice in CorpusAdvisoryMetadata.cs and CorpusGenerationPublication.cs:
- one deterministic format-v2
advisory-metadatarow per vulnerability id, strictly increasing by lexicographic UTF-8 byte order, matching PostgreSQL UTF8COLLATE "C", without case folding or Unicode normalization. Ill-formed surrogate input is refused rather than replacement-encoded. It carries explicit nullable display metadata, exact UTC round-trip timestamps, a finite one-decimal CVSS 2.0/3.0/3.1/4.0 tuple whose vector prefix matches its version, sorted distinct weaknesses/references, sorted document-digest origins,selectionVersion, and a verifiedsha256:metadata revision. Metadata arrays and each origin string tie-break use that same strict UTF-8 byte order. Any populated metadata requires at least one origin; the explicit all-null/empty row remains valid; metadataRevisioncovers every resolved field defined by the v2 profile, its selection version, and its ordered origins. Unknown v2 object properties are non-semantic extension metadata only; adding a resolved/revision-bearing property requires a new section format, which the v2 parser refuses;- the small
corpus.generation.completedv2 payload: opaque ownerartifactRef, manifest digest, real(sourceKey, sourceGeneration), and detached section descriptors(name, formatVersion, digest, bytes). The P6 envelope remains the only owner of epoch, sequence, and occurrence time. No section rows, storage bucket/key, or invented global generation enter the payload.artifactRefis exactlycorpus-publication:plus an 8–128-character ASCII alphanumeric/underscore/hyphen opaque token — never a URL, path, bucket/key, query, or fragment; - every known section name occurs exactly once in a v2 publication. Sorted distinct unknown extra descriptors are retained for forward compatibility;
advisory-metadatamust declare format 2, while the four legacy known sections use their real v1/v2 window. Required identity, descriptor order, digests, and those per-section rules fail closed.
FND-X18-2 completed the retained producer seam in source on 2026-08-21:
- Migration
002_fnd_x18_retained_corpus_publicationsadds the tenant-freefacts.advisory_metadata_inputcurrent projection and the smallruntime.corpus_publication*catalogue. Normalization writes metadata candidates, exact raw-document digest provenance, facts, and the document checkpoint in the same transaction.metadata-selection/v1considers only a candidate whose exact document digest still backs a live, non-tombstoned fact for the same vulnerability; authority/scalar fallback, atomic highest-score CVSS selection, array unions, and strictCOLLATE "C"tie-breaks are deterministic and ingestion time never participates. IngestSchedulerinvokesPublishCompletedGenerationAsynconly after a non-binary source generation completed and its normalization pump drained. The publisher streams all five sections from one repeatable-read snapshot, durably renames and verifies content-addressed files, then makes the catalogue pointer andcorpus.generation.completedv2 envelope visible atomically. A failure before readability leaves no handle, catalogue/current row, or v2 envelope.- Owner routes expose the current descriptor, manifest, and named sections at
/api/vulnerabilities/v1/corpus/publications/**. Handles remain opaque; manifest/part responses carry digest-backedETag,Digest, andX-StellaOps-Content-Digestheaders and support framework byte ranges/If-Range, so a disconnected multi-gigabyte transfer resumes without buffering it in the web process. Fetch contract, binding on any consumer routed through the gateway: section reads MUST be ranged, at 48 MiB or less per request. This is not a tuning preference.stellaops-router-gatewaymaterialises a whole upstream response into one transport frame, so an UNRANGED GET of a populated section (advisory-metadata is ~934 MB) fails with 502urn:stellaops:router:invalid-upstream-response— the frame is rejected, and neither failure mode names the size. Measured 2026-08-27 against the live gateway:<= 48 MiBranged -> 206 (48 MiB in 32.6 s), 64 MiB -> 504 after 30.3 s,>= 96 MiB-> 502 in ~1.8 s. Per-request cost rises faster than linearly with range size (48 MiB in one request took 32.6 s, while the same section read in 16 MiB requests averaged 2.2 s each), so a bigger range is not a cheaper range; 16 MiB is the measured comfortable size. The range machinery itself is correct end to end — the full 934 MB section was verified byte-for-byte via 59 x 16 MiB ranged reads in 132 s with zero failures, hashing to the catalogue’ssection_digest. A consumer that streams a section in one request works only against the in-network listener and fails in the deployed topology. - P16 defaults bound one publication to 16 GiB and retain exactly current + previous under a 32 GiB ceiling. The dated 2026-08-14 two-largest-section measurements (matcher 3.99 GB and consensus 5.73 GB) total 9.72 GB; the remaining sections must fit the residual budget or publication fails closed. Memory remains O(batch): one bounded streamed row and its pooled UTF-8 bytes plus fixed file/hash buffers, with metadata bounded to at most 256 candidates for the current vulnerability (kept in the declared selection order; further candidates are ignored with a logged warning, never a refused publication — 8 live vulnerabilities exceed the bound, measured 2026-08-26); only the five descriptors are retained in memory. The Findings consumer must fetch only its declared matcher, metadata, and evidence inputs, never
consensus-inputs.
The legacy in-memory v1 producer and unstamped /corpus/export route remain rollout compatibility surfaces. They still omit advisory-metadata and are not valid retained-generation/bootstrap authorities. The compose wiring was applied to the running estate on 2026-08-26 (FND-9 residual window): both hub roles run post-X18-2 images, migration 002 converged at boot, and the worker holds the RW publication volume. Two bring-up defects measured there are recorded with their fixes in vuln-ops.md: the volume root must be owned by the runtime uid, and stored fact identity strings must equal the trimmed identity VulnFact.ComputeFactId() hashes (migration 003_fact_identity_representation_convergence converges rows written before that guard).
The metadata projection is written for NEW documents only, so an estate that normalized before migration 002 publishes an all-null advisory-metadata section indefinitely (measured 2026-08-26: 0 candidates against 669,406 live vulnerabilities on a corpus whose connectors report “N seen, 0 new” per cycle). The owner-side backfill (sprint 003 VULN-B7) closes that: the worker-hosted CorpusAdvisoryMetadataBackfill walks each metadata-authority source’s retained ingest.raw_document payloads in the pump’s own (fetched_at, digest) order, at-or-before the source’s normalization checkpoint, only for documents whose digest still backs a live fact (the selector’s reachability predicate), and feeds them through the SAME normalizer and writer the pump uses — the result is a pure function of the retained document set, never of batch size or resumption. Its per-source checkpoint lives in facts.advisory_metadata_backfill (migration 004); the pass is default-off (Vulnerabilities:CorpusPublication:MetadataBackfill:Enabled), commits one batch per transaction, parks its checkpoint on the normalization head on completion, and a re-run changes nothing. Vulnerabilities known only through a non-authority source (alpine-secdb, debian-security-tracker) stay all-null by contract. Procedure: vuln-ops.md.
That pass has RUN on this estate (2026-08-26 15:08Z -> 2026-08-27 06:23:28Z; verified 08:30Z at 0388024ee9). facts.advisory_metadata_input holds 1,247,490 rows over 668,228 distinct vulnerabilities against 669,579 live vulnerabilities (99.798%), 0 of them unreachable by the selector’s live-provenance predicate, and the uncovered remainder is exactly the debian-security-tracker / alpine-secdb-only set that has no metadata authority — so coverage of the derivable population is 100%. The first publication after it carried a 979,698,214-byte advisory-metadata section where the last pre-backfill one carried 221,164,791 B of all-null rows. Re-verify with the coverage and reachability queries in vuln-ops.md step 3; the backfill is one-time and the pump is the standing mechanism from here, with FactWriter provenance now following the latest asserting document so a re-published record does not strand the selector on a stale digest.
Two bounds a consumer of that section has to respect, both measured 2026-08-27. Retention is a count, not a window: RetainedCorpusPublicationStore keeps the newest RetainedPublicationCount publications (live value 2, valid range 2-4) and prunes the rest inside the publishing transaction, with the CAS objects deleted immediately after commit. There is no age floor and no consumer-position floor. Under the pre-2026-08-27 cadence — one publication per completed source generation across 8 sources — the shortest measured artifact lifetime on this estate was 35 minutes. VULN-B8 (a) cut that to one publication per cycle, which multiplies the depth-2 grace by the number of enabled sources without changing a single retention value; (b) below is still open, so retention remains a blind count that consults no consumer. VULN-B8 © (2026-09-02) gave the pruned handle a typed signal: the prune is now a soft reclaim — the catalogue row survives as a time-bounded tombstone (reclaimed_at, migration 006; default window 14 days, Vulnerabilities:CorpusPublication:TombstoneRetention) while the bulk files stay GC-eligible (the CAS reference set excludes tombstones). The artifact routes answer HTTP 410 corpus_publication_reclaimed with reclaimedAt and the catch-up leg (GET /api/vulnerabilities/v1/corpus/publications/current) for a reclaimed handle, so it is no longer byte-identical to a handle that never existed (those still 404 corpus_publication_not_found / corpus_publication_section_not_found). GET /corpus/publications/current stays 200; the recovery path is unchanged, just now authoritatively signposted. Sharpest form of the problem: a full generation import measures 36 m 26 s while the shortest measured artifact survival under the old cadence was 35 m 25 s, so at RetainedPublicationCount = 2 no depth guaranteed a consumer could finish before its source was reclaimed — and HubEventMetrics.CorpusCatchUpLeg names publications/current as the declared recovery path for a consumer below the EVENT retention horizon, i.e. the protected plane’s recovery route points at the unprotected one. The once-per-cycle cadence makes the ordinary case comfortable; VULN-B8 (b) — owner-ruled to the HORIZON branch 2026-09-02 — completes the contract: GET /corpus/publications/current publishes an artifactRetention block (retainedCount, tombstoneRetentionSeconds, oldestRetainedSequence, retainedArtifactRefs[]). Membership of retainedArtifactRefs is the horizon test a consumer runs against the refs it holds; a held ref absent from the window answers 410 corpus_publication_reclaimed (while its tombstone lives) or 404 if it never existed. Producer deletion never depends on consumer state — deliberately, per the ruling: no lease cliff and no byte-ceiling/consumer-floor conflict can arise. The ranged-read requirement above remains the one open bound, recorded in sprint 003’s Decisions & Risks.
The publication catalog’s P13 class is cache-bounded: it holds generated corpus manifests and bounded reclaim tombstones, separate from the authoritative advisory/fact store and immutable release evidence. Forward migration 007_corpus_publication_retention_class.sql corrects the older bounded-current-previous comment without changing publication count/byte limits, tombstone windows or reclamation behavior. This source change does not imply a live migration has run.
7. Boundaries — who may reference what
src/ is the source of truth here and the rule is enforced, not documented: architecture conformance rejects every new cross-service source edge.
- Other services may source-reference
Contracts,Consensus, the DC-30VulnMatch.Coreartifact-consumer SDK, andBinaryArtifacts. These are classifiedcross-service-client-sdkinownership-manifest.jsonand are verified closed: their transitive graphs contain no service implementation. BinaryArtifacts contains local artifact computation and declared neutral analysis foundations; admission, persistence, evidence policy and signing/publication orchestration stay with the hub. A contract-like name is not the exemption; the verified graph is. - Everything else is a runtime seam: the owner API, versioned events, or the corpus artifact.
- No consumer compiles hub implementation any more. The shrink-only legacy-edge register reached zero pins on 2026-09-16 and is in zero-pin mode, so a cross-service implementation edge is a defect to fix rather than an edge to pin. (It previously carried consumers of this hub, each with an owning sprint and a sunset.)
8. Connectors
Tiering, the removal list, and the plugin contract are in connectors-architecture.md. The short version: Tier 0 is the credential-free default set that every estate gets; Tier 1 is opt-in (including the regional feeds); a removed connector can return as a Tier-1 plugin without a hub migration, because exploited-in-the-wild and friends are already modelled as fact attributes.
distro-binary-corpus is the DC-33 addition and rides the same SPI with no special casing. It streams distro repository index documents (deb-packages/rpm-repomd/apk-index) as binary-corpus raw documents; unpacking packages and extracting build-ids/fingerprints is the hub-side analysis stage, and package bytes are transient working data that never become ingest.raw_document rows. That split is a capacity invariant, not a preference — see design §9.1. The source is opt-in per estate, ships no default repository host, and fails closed when unconfigured; bring-up (mirror-first guidance, the capacity lever, and how to read the footprint check) is in the vulnerability operations runbook.
9. Related documents
- ADR-039 — database-per-service ownership
- vulnerability-data-plane-v2-design.md — target design, ERD, capacity budget, desk-check table, build/cutover plan
- Scanner advisory evidence identity — owner-item-13 hub-native identity transition (contract/readers/shadow adapter implemented; v2 production unimplemented)
- design-challenges-register.md — the DC-xx decisions cited throughout
- database-ownership-matrix.md
- Program sprints:
docs/implplan/SPRINT_20260722_*; running findings: the program ledger (agent coordination store)
