StellaOps.Vulnerabilities — module dossier
The exact-product storage and mirror delivered-order descriptions are source-verified against 7c5b3cb041eb5c9bbe48c681894cd8e4b1da7e4c (2026-10-21). Re-verify with the commands in product storage and mirror source verification. Full-seed and delta acceptance remain pending.
PCD-332 preserves repository-qualified OSV Composer product keys. Packagist ECOSYSTEM ranges, including repository-suffixed ecosystem names, use ordered concrete release bounds rather than generic-exact matching. The startup convergence rule re-normalizes the retained official Drupal documents whose earlier facts have the wrong scheme; the qualified product identity stays the same while the scheme-dependent fact identity is corrected through normal revisions and tombstones. Missing retained documents are reported rather than silently retired. Scanner must supply repository provenance; an unqualified component is not aliased to an explicitly different repository. That rule selects only official Drupal repository keys. PCD-823’s osv-packagist-scheme converges every other OSV Composer fact still stored as generic-exact (plain Packagist and Drupal-namespace keys without a repository qualifier) the same way, from the retained documents.
PCD-332 source verified against
1e52c13050(2026-10-06). Live convergence verified on 2026-10-10 (read-only) against the Hub Worker built fromc85f15158a: checkpointvuln.fact-key-convergence#osv-drupal-packagist-scheme#osvat 2026-10-09 15:59:31Z, 755 livesemverand 0 livegeneric-exactofficial Drupal repository-qualified OSV facts. Re-verify:pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Facts.Tests/StellaOps.Vulnerabilities.Facts.Tests.csproj -All -BuildProjectReferences -MaxThreads 1.
PCD-823 source verified against
8cf026030c(2026-10-11): ruleosv-packagist-schemeand the pump-order replay. Its live convergence follows a Hub Worker deploy from a main that contains it. Re-verify:pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Facts.Tests/StellaOps.Vulnerabilities.Facts.Tests.csproj -Method '*PackagistScheme_*','*ReplaysRevisionsOldestFirst*' -BuildProjectReferences.
PCD-823 metadata guard verified against
8db02cd8dc(2026-10-11): a convergence replay never moves an advisory metadata candidate to an older revision of its source record. Re-measured read-only on the local hub: theosv-packagist-schemereplay writes 7,187 candidates and changes none (without the guard, 497 rows would have moved back). Re-verify:pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Facts.Tests/StellaOps.Vulnerabilities.Facts.Tests.csproj -Method '*Replay_Of*','*UbuntuUpstreamCve*' -BuildProjectReferences.
PCD-826 source verified against
28ed238ee9(2026-10-11): every advisory metadata writer keeps the newest retained revision except the mirror seed, and the default-off revision repair pass. Its live repair follows a Hub Worker deploy from a main that contains it. Re-verify:pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Facts.Tests/StellaOps.Vulnerabilities.Facts.Tests.csproj -Class 'StellaOps.Vulnerabilities.Facts.Tests.CorpusAdvisoryMetadataRevisionOrderTests' -BuildProjectReferences.
Verified against commit
942a18727con 2026-09-26 (UTC). Re-verify:git grep -n "MapGroup\|MapGet\|MapPost" -- src/Vulnerabilities/StellaOps.Vulnerabilities.WebServiceandsed -n '/^ vulnerabilities-web:/,/^networks:/p' devops/release/bundle/docker-compose.yml.
The target design and the build record of the consolidation live in vulnerability-data-plane-v2-design.md. This dossier states what src/Vulnerabilities/ implements today.
1. What this service is
The vulnerability knowledge hub: one microservice owning one database (stellaops_vuln) that holds raw and normalized vulnerability, VEX, distro, EPSS, binary and symbol knowledge for the whole platform. It replaced the Concelier (including Feedser and Excititor), VexHub and VexLens services; those sources are frozen under src/__Obsoleted/Concelier/ and their dossiers are archived (section 14).
Two properties define it:
It is tenant-free. There are no tenant columns, no row-level security, and no per-tenant gate decisions anywhere in stellaops_vuln. What the world says about a vulnerability is not a per-customer fact. Everything tenant-scoped (a tenant’s own VEX decisions, source-trust overrides, gate verdicts) lives in the control plane (Policy, Findings).
It is rebuildable from its sources. Every derived table can be dropped and reconstructed from ingest.raw_document plus a re-ingest. That is the documented disaster recovery path.
2. Deployables
| Deployable | Host project | Compose | Release bundle |
|---|---|---|---|
vulnerabilities-web | src/Vulnerabilities/StellaOps.Vulnerabilities.WebService | devops/compose/docker-compose.vulnerabilities.yml | yes, image stellaops/vulnerabilities-web |
vulnerabilities-worker | src/Vulnerabilities/StellaOps.Vulnerabilities.Worker | devops/compose/docker-compose.vulnerabilities.yml | yes, image stellaops/vulnerabilities-worker |
- The web host registers with the Router as microservice
vulnerabilities-web(AddRouterMicroserviceinProgram.cs) and carries the network aliasesvulnerabilities.stella-ops.localandvulnerabilities-web.stella-ops.localin the bundle. - The worker hosts the ingest scheduler (
IngestSchedulerWorker), the consensus projector (ConsensusProjectionWorker), mirror-seed publication (MirrorSeedPublicationWorker), binary corpus analysis (BinaryCorpusAnalysisWorker), the advisory-metadata backfill (AdvisoryMetadataBackfillWorker), the advisory-metadata revision repair (AdvisoryMetadataRevisionRepairWorker, default off) and the outbox/inbox retention pruners (src/Vulnerabilities/StellaOps.Vulnerabilities.Worker/Program.cs). - Both roles share the
vulnerabilities-corpus-publicationsvolume. The worker mounts it read-write and is the only publication writer; the web role mounts it read-only. - In the release bundle the ingest scheduler stays at its default (
Enabled = false,Vulnerabilities:Ingestion:Scheduler), so upstream feed connectors do not run. The bundle’s data arrives through the signed mirror seed that the installer imports (section 9). - Restart cadence (PCD-349). The scheduler’s singleton
runtime.ingest_cycle_statestores the last fully attempted cycle’s completion time. A restart waits out the remaining configured interval; a cycle contains per-source failures as before. Cancellation or a whole-cycle exception does not advance that completion cursor. A session fence serializes worker cycles, and connections/scopes are released during the wait. At startup, including disabled scheduling, recovery abandons active generations only after acquiring their existing source fence. A still-fenced writer is preserved, as are accounting, completed generations and feed resume cursors. Migration 016 creates the empty one-row-bounded cadence store; it does not seed a completion timestamp or fixture data. Verified against80f2f9dc93; reverify withpwsh tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Ingestion.Tests/StellaOps.Vulnerabilities.Ingestion.Tests.csproj -Method '*IngestSchedulerTests.Worker*' -BuildProjectReferences -Restore.
3. Database
| Item | Value |
|---|---|
| Database | stellaops_vuln on the installation PostgreSQL server |
| Role | vuln (devops/release/bundle/.env.example) |
| Connection | STELLAOPS_POSTGRES_VULN_CONNECTION only; no shared-connection fallback (VulnerabilitiesPersistenceExtensions.ConnectionEnvironmentVariable) |
| Migrations | embedded SQL in src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Persistence/Migrations/, applied on startup through AddStartupMigrations; forward-only |
The schemas are organizational, not ownership boundaries: ingest, facts, consensus, dist, runtime, binaries and symbols (section 5). Migration 012_retire_unused_hub_tables.sql dropped the declared-but-unused tables, and 013_hub_retention_classes.sql records the retention class of each remaining table. Migration 020_retire_dist_statement.sql dropped dist.statement, which repeated the live facts (section 5). The retention and custody procedure is the table-by-table record.
The owner registers the heavy indexes in migrations 014 and 022 as deferred obligations (DIP-21). Startup executes the remaining immutable SQL and records the obligation beside the original migration checksum. After host start, the shared builder creates the indexes concurrently in this same database. runtime.schema_deferred_indexes has one cache-bounded row per registered index; retries overwrite that row. The identity is (index_schema, index_name), and the pinned migration and contract checksum bind its meaning. The Hub currently declares two rows, with no document payload or per-attempt history. This cache class is outside the source/derived data-identity register. The builder has no cross-service database access.
Corpus streaming checks the exact corpus-order index before scanning; tombstone reclamation waits for its exact tombstone index. Web and worker health readiness remain false until both indexes are valid. Corpus readiness reports rebuilding until then and subsequently applies its generation-based content check; source-generation rows remain visible. Missing or invalid indexes never authorize an expensive fallback scan. See deferred index operation for configuration, retry and recovery semantics. Source validation does not establish live rollout.
4. Source layout
| Project | Role |
|---|---|
StellaOps.Vulnerabilities.Contracts | Closed contracts SDK (DC-25). Zero ProjectReferences by contract; purity is conformance-verified, not assumed. |
StellaOps.Vulnerabilities.Consensus | THE consensus evaluator (DC-38) — one pure function shared by the hub (global consensus) and Policy (tenant-effective consensus). ConsensusApplicability derives the effective affected range (the winning statements’ bounds) for Policy’s gate projection and Scanner’s hub-backed matcher. Also a classified client SDK, for the same reason: it is pure. |
StellaOps.VulnMatch.Core | DC-30 artifact-consumer SDK. The one deterministic, offline-capable SBOM-to-advisory matcher used by online and offline consumers. The assembly/namespace contract is preserved; its closed graph reaches only neutral version comparison plus the pinned Semver package and contains no database provider, HTTP, DI, host, clock, or configuration binding. |
StellaOps.Vulnerabilities.RawModels / .Normalization / .Canonical | Ingestion substrate carried over from the v1 tree where the transfer was purely mechanical: raw document shapes, version-range normalization, canonical domain primitives. |
StellaOps.Vulnerabilities.Persistence | The fresh schema baseline + startup migration wiring. Auto-migrates the service’s OWN database; fails closed without STELLAOPS_POSTGRES_VULN_CONNECTION. |
StellaOps.Vulnerabilities.Ingestion | Connector SPI, the ingest pipeline, the capacity guard, and first-party issuer submission. |
StellaOps.Vulnerabilities.Facts | The fact store, projectors, EPSS, mirror import/export, the corpus artifact, and the serving queries. |
StellaOps.Vulnerabilities.BinaryArtifacts | Closed offline SDK for native-code normalization, signature authoring/matching and predicate formats. Shared by the CLI and hub; no reference admission, persistence or issuer workflow. |
StellaOps.Vulnerabilities.BinaryAnalysis / .Binaries | Digest-bound reference generation, bounded package analysis, exact binary subjects and owner persistence. |
StellaOps.Vulnerabilities.Fingerprints / .DeltaSig | Hub-owned feature generators, feature storage, evidence policy and Symbols contract adaptation. |
StellaOps.Vulnerabilities.Symbols / .Symbols.Client | Global symbol source, catalog and manifest stores (SymbolManagementStore, SymbolManifestStore); .Symbols.Client is a classified closed client seam. |
StellaOps.Vulnerabilities.WebService / .Worker | The two deployables (vulnerabilities-web, vulnerabilities-worker). |
Deployable images and their plugin payloads
vulnerabilities-web is a Router microservice over Valkey messaging. It needs three signed payloads, at the paths the bundle configuration names:
| Payload | Path in the image | Configuration key |
|---|---|---|
| Messaging backend bundle | /app/plugins/messaging/valkey/ | Router__Messaging__PluginDirectory=/app/plugins/messaging |
| Router messaging transport bundle | /app/plugins/router/transports/router-transport-messaging/ | Router__TransportPlugins__Directory |
| Trust roots | /app/trust-roots/plugins/messaging/cosign.pub, /app/trust-roots/plugins/router/ | Router__Messaging__TrustRootPath, Router__TransportPlugins__TrustRootDirectory |
The hub scans /app/plugins/messaging and finds the valkey bundle one level below it. Most other hosts point the directory at the bundle itself, /app/plugins/messaging/base/valkey. A payload tree copied out of another service’s image therefore does not load in the hub.
devops/docker/build-service-publish.sh builds the image without payloads. The release build adds them as a separate layer (devops/release/plugin-layers/services/vulnerabilities-web.Dockerfile, applied by devops/release/bundle/tools/build-plugin-layers.sh). Only the layered image starts in the bundle. Check a candidate image before starting or publishing it:
python tools/scripts/validate/check-bundle-messaging-payload.py --image <image-ref>
vulnerabilities-worker is not a Router host and has no plugin layer.
The family solution (StellaOps.Vulnerabilities.slnx) deliberately lists only family-owned projects. dotnet slnx auto-add pulls the whole transitive closure — trim it, or the solution stops being evidence of anything.
5. Data model
ingest—raw_document(content-addressed originals),source_stateandnormalize_checkpoint. Normalization reads the retained originals directly. A document’s facts are written in one transaction of a few set-based statements (FactWriter), each with a 300 s command timeout of its own: a large Red Hat CSAF document’s provenance insert outlasted the 30 s driver default on a 166 GB hub.facts—fact_current, the exactproduct_key_dictionary, thefact_provenanceassociations and immutableprovenance_origindictionary, andfact_revision(range-partitioned history, migration011_fact_revision_range_partitioning.sql),fact_identity_map(the old-key to new-key mapping a key-convergence re-key records, migration023_fact_identity_map_and_revision_prune.sql), the advisory-metadata inputs and backfill position, exploit evidence, and the EPSS tables. The ordinaryfact_provenance_readjoin view returns all seven original logical values; provenance storage describes exact references, custody and startup conversion.fact_current_readreconstructs the complete fact including its literal product. The internal product reference changes no domain hash or public cursor; see product storage.consensus—issue_linkset(global and narrow: counts, status/severity summaries, content hash), with exact products reconstructed byissue_linkset_read.ConsensusProjectorconsumes thevuln.factsstream and emitsvuln.consensus, which Policy consumes.dist— no table. The VEX statements (the former VexHub statements) are computed on read fromfacts.fact_current_readbyHubQueries.GetVexStatementsAsync: oneopenvexstatement per live fact with provenance, with statement idsha256(fact_id || 'dist')and ajsonb_build_objectof vulnerability, product, status, justification and applicability, ordered by statement id. Until SPRINT_20261007_063 HSR-1 the consensus projector stored the same values indist.statement(15 GB, 24.1M rows on the dev estate on 2026-10-07); migration020_retire_dist_statement.sqldrops that table. The route now also reflects a fact change before the projector drains it.runtime—source_generation, the retained corpus publication catalogue, its bounded resume epoch and mutation witnesses, the superseded-identity reclamation receipts (superseded_identity_reclaim_receipt, migration022_superseded_identity_reclamation.sql), the revision prune receipts (superseded_revision_prune_receipt, migration 023), and the singleton provenance conversion/GC cursor and completion receipt (provenance_origin_conversion, migration 030) and product conversion/verification/GC cursor (product_storage_conversion, migration 031).eventing— completeoutbox_batchenvelopes, inbox, checkpoints and leases, all in this same owner database. See event storage for logical replay and retention.binaries— the binary-granularity plane: binary identity, package maps, fingerprint and delta-signature features and build provenance. Distribution fix thresholds and their retained evidence derive from the hub facts and provenance through the distribution evidence contract. A binary fact is the resolution index from a binary applicability scheme (buildid/fingerprint) ontofacts.fact_current, not a verdict store.symbols— the symbol CAS catalog (sources, catalog entries, manifests, blob catalog). Blob bytes live in the object store, never in this database.
The identity rules worth knowing before you touch anything
A fact’s identity includes its applicability, and a revision hash is a value, not an identity. Two statements about the same (vulnerability, product) with disjoint version ranges are two distinct facts and must never collapse into one — that collapse is what flattened the v1 corpus. When a vendor republishes the same statement with a changed status, the fact keeps its identity and gains a revision; the old revision stays in history.
Ubuntu OSV records named UBUNTU-CVE-<year>-<number> aggregate under the corresponding CVE-<year>-<number> only when their upstream array explicitly confirms that exact CVE. The Ubuntu record remains the source statement and provenance reference; its package, release, range and issuer remain independent. Existing CVE aliases retain precedence. Other upstream IDs do not become aliases: OSV’s upstream relationship is directed and can refer to several vulnerabilities. A missing or unrelated upstream CVE leaves the Ubuntu record identity unchanged.
fixedVersion is the declared fix. Applicability fixed is an exclusive upper bound; lastAffected is an inclusive upper bound and never a fix (PCD-334). OSV last_affected and mirror lastAffectedVersion map to the latter while leaving the declared fix null. No successor version is invented. Consumers that need a real fix, such as Scanner’s surface feed, read fixedVersion. If a source states both upper constraints, matching applies both.
NVD versionStartIncluding maps to inclusive introduced; versionStartExcluding maps to introducedExclusive. End-excluding remains fixed, and end-including is lastAffected with no declared fix. Exact CPE versions remain literal generic-exact; unbounded CPE claims remain product-level. Ordered NVD claims retain generic-range, never a per-record semver inference. The neutral comparator supports ASCII letters/digits and ., -, _, + separators, strips numeric leading zeros without integer conversion, and treats missing trailing zero tokens equally (1.0 equals 1.0.0). Wildcards, operators and other grammar are unsupported for ordered bounds. Literal exact claims compare the complete source value ordinally (01 differs from 1).
The nvd-endpoint-inclusion startup rule first qualifies vulnerable, resolvable raw CPE matches with start-excluding/end-including markers, then applies strict whole-document admission under the existing owner fence. The retained lower identity slot is Introduced ?? IntroducedExclusive; corrected meaning changes revisions and provenance. Unrelated malformed exclusive-only documents remain outside this correction, and unavailable raw inputs remain unclassifiable residuals. Mirror export/re-import retains declared generic comparison and refuses ambiguous same-ID meanings.
Scoped retained identity (SPRINT_20261007_063 HSR-16, owner ruling Q-47). A CVE or NVD statement is keyed by its coarse product key (pkg:generic/cpe/{vendor}/{product}, pkg:generic/cve/{vendor}/{product}). When two statements of one document share a coarse fact id with distinct meanings, each statement of that id is keyed by its scope instead, as purl qualifiers in key order: the CPE edition, sw_edition, target_sw and target_hw (ANY and NA add nothing), and the platform context. For NVD the platform is the set of non-vulnerable CPEs AND-ed with the match (part:vendor:product[:version], a negated node’s CPE prefixed with a raw !, sorted and comma-joined; a list longer than 256 characters becomes sha256:<64 hex> of that list, so the key stays far below the index tuple limit); for CVE it is the entry’s platforms. Examples: pkg:generic/cpe/gitlab/gitlab?sw_edition=community, pkg:generic/cpe/microsoft/windows_10_1607?target_hw=x86, pkg:generic/cpe/cisco/ios_xr?platform=h:cisco:asr_9001, pkg:generic/cve/acme/robot_1?platform=h0.1.7. Values are percent-encoded with : and / raw, as CSAF qualifiers are spelled. A statement whose coarse id carries one meaning keeps it, so the ids of all other stored facts do not move. The result depends only on the set of statements: no range union, no position index, no last-wins choice. Statements that still share an id with distinct meanings are refused before any write, per identity (see “Intake refuses per identity” below): that id is not written and the document’s other statements are; on the hub that is one identity of one NVD document (CVE-2023-48795, the same CPE claiming both end inclusions). The CVE normalizer derives a CPE-keyed entry’s attributes from the same CPE, so a CVE entry and an NVD match with the same CPE and scope share one key.
PCD-445 source verified against
56357b8682799c029045a161491eabab422373e8(2026-10-08 UTC). Re-verify:pwsh tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Facts.Tests/StellaOps.Vulnerabilities.Facts.Tests.csproj -Class "*NvdProducerBoundTests" -BuildProjectReferencesand the same project with-Method "*NvdEndpoints*" -BuildProjectReferences.
CVE 5.x lessThan maps to exclusive fixed and keeps fixedVersion; lessThanOrEqual maps to inclusive lastAffected and leaves fixedVersion null (PCD-444). The normalizer refuses a version object with conflicting upper fields or an explicitly malformed or blank endpoint value, and an affected entry’s defaultStatus never widens that statement to a product-level claim. Equivalent repeated claims deduplicate without adding a product-level fallback. Different meanings sharing one retained fact ID within the same source document are keyed by their scope (CPE edition and target attributes, platforms; see “Scoped retained identity” above) and refused before writes only when the scope cannot tell them apart; source order cannot choose meaning.
Intake refuses per identity (SPRINT_20261007_063 HSR-17, lead decision Q-48). A CVE or NVD statement that cannot be retained is refused as one identity (vulnerability and product key), not as a document: an unreadable or conflicting bound (invalid_version_bound, conflicting_version_bounds, NVD conflicting_version_carriers) and distinct meanings that no scope tells apart (ambiguous_applicability_identity). The normalizer reports the refused identity instead of throwing and returns the document’s other statements. NormalizationPump refuses them through its standing intake guard, the same path as a fact whose vulnerability id is malformed (VULN-B9): the identity is not written, the document’s other facts and advisory metadata commit with the checkpoint, and the worker logs a warning with the source, document digest, vulnerability, product key and reason. The pump’s summary line and NormalizationPumpResult.RefusedIdentities count the refused identities. A fact an older normalizer stored for a refused identity stays as stored. Before this, one such document rolled back its transaction before the checkpoint advanced, and the next cycle read it first again, so the source stopped normalizing until the document changed. A document the normalizer cannot read at all (malformed JSON, a record without its id) still fails as a whole. Key convergence applies the same rule (see “Key corrections converge on worker start”).
The cve-less-than-or-equal startup rule reads legacy bounded CVE candidates under the existing source normalization fence. It qualifies retained payloads with the corrected CVE normalizer and replays only documents that actually contain an inclusive claim. Exact and exclusive-only documents remain unchanged. Raw qualification first checks affected, non-git inclusive markers on resolvable products, so newly strict admission cannot block replay on an unrelated historical exclusive-only document. Actual inclusive candidates still undergo full strict normalization. Corrected facts retain identity and append a changed revision through the existing writer/provenance transaction; the completion checkpoint makes the second pass a no-op. Missing raw payloads retain their facts and count as unclassifiable bounded candidates, because the collapsed stored shape cannot distinguish inclusive from exclusive input.
Source boundary tests feed the normalizer and actual serving/corpus/mirror output to the Findings ordered matcher using the exact fact product key. They prove endpoint meaning, not discovery of generic CPE products from a scanned component. Generic product discovery retains the explicit limitations in the product-key table below. Live adoption requires a separately reviewed consumer-before-producer deployment seal; PCD-337’s earlier pinned candidate does not include this producer correction automatically.
PCD-444 source verified against
87feffb908(2026-10-07 UTC). Re-verify:pwsh tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Facts.Tests/StellaOps.Vulnerabilities.Facts.Tests.csproj -Class "*CveProducerBoundTests" -BuildProjectReferencesand the same project with-Method "*CveLessThanOrEqual*" -BuildProjectReferences.
Correcting an inclusive value previously stored in fixed preserves its fact ID: the identity hash retains the original upper-value slot (Fixed ?? LastAffected). Storage and revision hashes retain the distinct meaning, so the normal writer appends a changed revision and publishes it without retiring the same fact. Startup convergence re-normalizes retained OSV and mirror documents that still contain the old representation. A mirror document whose earlier export already mislabeled the value as a fix needs a fresh corrected export; missing meaning is not inferred from that document.
A statement carrying both exclusive and inclusive upper constraints includes the inclusive constraint separately in its identity; two different ranges cannot collapse because their exclusive fix agrees. This does not change the identity of corrected last-affected-only facts. The mirror replay also visits bounded statements with a declared fix once, because retained documents may carry both constraints. Fixed-only facts keep their identity and revision; restoring a previously omitted extra constraint creates the distinct range and retires its superseded fact through the existing provenance transaction.
PCD-334 source verified against
8e65746d01(2026-10-06). Re-verify:pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Facts.Tests/StellaOps.Vulnerabilities.Facts.Tests.csproj -Method "*LastAffected*" -BuildProjectReferences -MaxThreads 1.
Provenance is many-to-one, and only the last origin’s removal tombstones a fact. The same statement arriving via a connector claim and via a derived distribution is ONE fact with TWO provenance rows. Removing one origin leaves the fact alone. Removing the last one tombstones it and rebuilds its linkset in the same transaction — there is no retention sweep in v2, and adding one would reintroduce the deletion-by-schedule behaviour the v2 design removed.
An issue linkset’s subject identity is (vulnerabilityId, productKey); its content hash is an observation revision. This is the consensus.issue_linkset primary key, with no tenant dimension. The values are ordinal and case-sensitive — especially productKey, where mixed-case Maven group IDs are ordinary — and consumers must preserve the exact producer-returned case. contentHash is SHA-256 over the canonical projection including live fact revision hashes, so a value change keeps the subject and changes its observation revision. Scanner’s worked evidence contract applies that distinction in advisory-evidence-identity.md; it never derives a Guid. Product-key path routes recover the final segment from Kestrel’s raw request target and decode HTTP transport exactly once. This preserves canonical PURL escapes such as OCI-name %2F and digest %3A; applying UnescapeDataString again to the mixed bound route value changes the producer-owned identity and is forbidden.
The product-key contract is producer-owned: ProductKeyV1in src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Contracts/AdvisoryEvidence/ProductKeyV1.cs. The hub’s normalizers define the key shape, so the hub publishes the derivation: a consumer turns a component purl into the lookup key for GET /api/vulnerabilities/v1/products/{productKey}/issues with ProductKeyV1.TryDeriveLookupKey, and the Alpine and Debian normalizers write their keys through ForApkSource / ForDebSource. The key never carries a version.
| Purl type | Hub key | Source of the name | Written by |
|---|---|---|---|
apk | pkg:apk/{distro}/{source} | origin qualifier, else upstream, else the package name; decoded, no qualifiers | Alpine secdb; OSV derives the same key through ProductKeyV1 |
deb | pkg:deb/{distro}/{source} | same rule; the Debian tracker keys source packages | Debian security tracker; OSV (Ubuntu and other Debian-family purls) derives the same key through ProductKeyV1; the release series is the applicability platform, not part of the key |
rpm | pkg:rpm/{vendor}/{name}?arch={arch}[&epoch={epoch}] | the binary name, percent-encoded as the vendor spells it; epoch from the epoch qualifier or an N: version prefix, epoch 0 omitted | Red Hat CSAF; SUSE CSAF package filenames with a known distro release and architecture |
generic | none: TryDeriveLookupKey returns false and the consumer asks nothing | the hub’s generic facts sit under issuer-invented namespaces (pkg:generic/cpe/…, cve/…, suse-csaf/…, redhat-csaf/…, osv/…) or an exact vendor file checksum, which no component purl carries | NVD, CVE, SUSE and Red Hat CSAF fallbacks |
| any other | the purl without its version | qualifiers and subpath kept exactly, no case folding; the CSAF normalizer alone stores : and / in qualifier values unescaped, because Red Hat writes one product both ways (checksum=sha256%3A… and checksum=sha256:…) | OSV, mirror, CSAF, OpenVEX |
Consequences worth knowing: subpackages of one source (libssl3, libcrypto3) share one Alpine or Debian key; a distro purl without a namespace has no distro identity and matches no distro-keyed issue; Red Hat modular keys (rpmmod=) cannot be derived from an installed-package purl; a scan whose components are all pkg:generic (for example unresolved native binaries only) has no product keys, so its hub-backed report is unavailable with reason hub-product-issues-no-product-keys rather than a clean zero, and Red Hat CSAF file products keyed by checksum (pkg:generic/<file>?checksum=sha256:<hex>; 408 distinct keys on a seeded hub, 2026-09-29) are not reachable from a scan. The shared fixture src/__Tests/corpus/product-key-contract-v1/cases.json pins both sides: the Contracts tests derive every case, the Facts tests run each hub normalizer on it, and the Scanner worker tests run the Scanner’s purl builders and hub adapter on it.
A distro fact carries the release it applies to: DistroPlatformV1in src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Contracts/AdvisoryEvidence/DistroPlatformV1.cs. A distro feed states a fix per release, while the product key names only the source package, so without the release a newer release’s range, or an unfixed entry from any release, would mark an older release’s fixed package as affected (measured before the change: a fixed bookworm openssl inside trixie’s range; a fixed noble openssl inside the Ubuntu Pro and FIPS [0,∞) entries). The normalizers write applicability.platform, and consumers scope an issue’s statements to the installed package’s platform before they evaluate consensus.
| Platform | Grammar | Feed side | Image side |
|---|---|---|---|
| Debian | debian:<codename> (debian:bookworm, debian:sid) | the tracker’s release name; OSV Debian:12 through a fixed number-to-codename table | os-release VERSION_CODENAME, else VERSION_ID through the same table; SBOM purl distro=debian-12 |
| Ubuntu | ubuntu:<VERSION_ID>[:<stream>] (ubuntu:22.04, ubuntu:22.04:fips-updates) | the OSV ecosystem (Ubuntu:22.04:LTS, Ubuntu:Pro:FIPS-updates:22.04:LTS); Pro and the ESM pockets fold into the base release, other pockets stay a stream (fips, fips-updates, fips-preview, realtime, nvidia-bluefield) | os-release VERSION_ID; SBOM purl distro=ubuntu-22.04 |
| Alpine | alpine:<major.minor> or alpine:edge | secdb distroversion (v3.19); OSV Alpine:v3.19 | os-release VERSION_ID (3.19.1; an _alpha build is edge); SBOM purl distro=alpine-3.19.1 |
| Red Hat Enterprise Linux | rhel:<major> (rhel:9); exclusive streams rhel:<major>:<kind>-<major>.<minor> for eus, aus, e4s, tus, e2s (rhel:9:eus-9.4) and rhel:<major>:els | the CSAF platform product’s CPE (cpe:/o:redhat:enterprise_linux:9::baseos, cpe:/a:redhat:rhel_eus:9.4::baseos), else its stream id (BaseOS-9.6.0.Z.MAIN.EUS, .GA and .Z.MAIN are the major; .Z.EUS, .Z.AUS, .Z.E4S, .Z.TUS, .Z.E2S the minor stream; 7Server, 4AS the major; an ELS segment els); layered products (Satellite, OpenShift, JBoss EAP) name none | os-release ID=rhel (UBI included), VERSION_ID major; Red Hat’s rpm namespace pkg:rpm/redhat/ |
| SUSE | sles:<major>[.<sp>], sled:<major>[.<sp>], opensuse-leap:<major.minor> | the CSAF platform product’s CPE (cpe:/o:suse:sles:15:sp5, the sle-module-* CPEs of that release) or name (SUSE Linux Enterprise Server 15 SP5); package filenames map to RPM keys in the corresponding sles, sled or opensuse-leap namespace, with architecture and nonzero epoch retained | os-release ID and VERSION_ID (15.5) |
DistroPlatformV1.Applies is the one rule: a fact without a platform applies everywhere, every fact applies when the image platform is unknown (the behaviour before platforms existed), otherwise the ID and release must match, and on Ubuntu a FIPS stream applies exactly to FIPS installs (the installed version or the package name says fips, as in 3.0.2-0ubuntu1.17+Fips1 or linux-fips). FIPS streams are an Ubuntu concept: on any other distro a package whose name or version says fips (fipscheck, dracut-fips) keeps its release’s facts. A Red Hat minor-release stream (EUS, AUS, E4S, TUS, E2S) or Extended Life Support stream applies only to an install that names the same stream, which no os-release does, so those statements stay off images (an EUS-pinned host is judged by the base stream’s statements). Other streams apply like their base release. The platform is part of the applicability hash only when it is set, so non-distro facts keep their fact ids; a platform on a binary applicability is refused. Where it travels: the issue-facts route (applicability.platform), the corpus consensus-inputs rows (additive, omitted when absent), matcher-rows v5 (one row per release), mirror-seed claims and the T9 mirror export (platform on each version range, and a release-scoped fact without bounds as a range with its platform only), so a seeded install writes the same scoped facts. The Scanner records the image platform on its deb, apk and supported rpm OS packages as the SBOM property stellaops.os.platform (read from the rootfs os-release; the purl is unchanged) and falls back to the purl distro qualifier for an external SBOM. DistroPlatformV1.ComponentPlatform is that one rule for every consumer: the image platform for an OS package whose purl namespace is the image’s distro (pkg:deb/<id>/, pkg:apk/<id>/, pkg:rpm/redhat/ on rhel, and the exact pkg:rpm/sles/, pkg:rpm/sled/ or pkg:rpm/opensuse-leap/ namespace on its corresponding SUSE image), else the purl distro qualifier. Consumers that scope by it: the Scanner’s hub stage, Policy’s gate projection (one row per release), the Findings matcher (matcher-rows v5 and the SBOM component’s release) and the CLI offline database (compact schema v8, including repository-qualified identity). The shared fixture src/__Tests/corpus/distro-platform-v1/cases.json pins the Contracts rule, each hub normalizer and each consumer’s decision.
Release coverage: “release not covered” instead of a clean zero (DistroReleaseCoverageV1 in src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Contracts/AdvisoryEvidence/DistroReleaseCoverageV1.cs). Because a distro fact applies only to the release it names, an image installed from a release that no ingested feed covers receives no facts. The Alpine secdb normalizer writes one platform per configured secdb branch, so a hub that ingests the v3.19, v3.20, v3.21, v3.22 and edge branches gives an Alpine 3.24 or 3.12 image no apk facts, which reads as zero findings. The rule tells that zero apart from a clean result, over the live statements the hub returns for the image’s own packages, the inputs every consumer already reads: the component’s release (DistroPlatformV1.ReleaseOf, <os-release ID>:<release> with any stream folded into its release) is covered when one of those statements names it, not covered when they name other releases of the same distro and none names it, and undetermined when they name no release of that distro. A not-covered release is reported with the reason release-not-covered:<release> (release-not-covered:alpine:3.24), never as a clean zero. Coverage is decided over the whole image, not one package: per-release feeds state every package’s history on each release they cover (an Alpine secdb branch lists the secfixes of every package in that branch, the Debian tracker each release per source package), so a package whose first fix landed after a branch fork does not make a covered release look uncovered. Limits: a distro that no ingested feed covers at all, and an image whose packages carry no history on any release, are undetermined, not “not covered”. In the other direction, the premise holds only for feeds that publish per-release history. An OSV distro ecosystem (Ubuntu:<release>, Debian:<n>, Alpine:v<n>) names only the releases a record affects: the OSV normalizer writes one platform per affected[] entry. An image on a covered release whose packages received statements only for other releases of its distro therefore reads not covered, a false release-not-covered:<release>. That error fails closed (the source is unavailable and a blocking environment’s reachability gate denies), and it is unlikely for a typical image because coverage is decided over all of its packages; DistroReleaseCoverageV1ContractTests pins the behaviour. Consumers: the Scanner’s hub stage (its v2 report is unavailable with that reason; see the Scanner dossier).
6. Reconciliation protocol
Every projection in this service follows the same four rules, and they are not independently negotiable:
- One source transaction commits the fact revision/tombstone and its ordered outbox row. Absence-tombstones are authorized only by a completed generation — a connector that died halfway through must not be able to delete what it simply failed to fetch.
- One leased writer per stream, with a fencing token checked INSIDE the destination transaction. A second instance blocks on the lease; an expired holder’s commit is rejected by its stale token rather than by hoping it noticed in time.
- One destination transaction per drained batch: admit every envelope of the batch → rebuild each distinct affected linkset once, from current facts → advance the checkpoint to the batch’s last envelope. A failure rolls back all of it. A checkpoint that could advance past a failed derivation would silently drop the event forever. The consensus projector emits one
consensus.changedper linkset that moved in the batch, committed with the change. - Replay is physically idempotent. Content-guarded upserts mean a replay writes zero hot-row versions — asserted directly by summing
xminbefore and after a full re-consumption.
Trap, learned the hard way (2026-08-02): the producer-epoch check must run before any empty-batch early return. A rebuilt producer restarts its sequence at 1, so a stale checkpoint of (old epoch, 7) filters seq > 7 against the new epoch and legitimately returns zero envelopes. An early return there leaves the consumer permanently frozen on a stream that is producing normally. An empty batch is not evidence that your checkpoint is still valid.
Projection lease verification
ConsensusProjectionHostTests verifies renewal during a drain with controlled batch time and a logical lease expiry, while PostgreSQL still supplies acquisitions, renewals and transaction fencing tokens. The healthy drain must outlast its logical TTL, keep one token and reach zero lag; a planted batch that outlasts that TTL must produce a self-fence and re-acquisition. The independent takeover test expires the real PostgreSQL lease, acquires it with a competing holder and verifies that the incumbent stops until the competitor releases it.
Consensus projection throughput
The fact writer emits one vuln.fact.upserted envelope per changed fact, so a linkset with k facts arrives as k adjacent envelopes. Each rebuild reads all k live facts (until HSR-1 it also re-checked their k dist rows). Before SPRINT_20260928_002 (PCD-111) the projector committed once per envelope and rebuilt the whole linkset each time. On the local estate that cost 8 ms + 0.14 ms × k per event, measured from the gaps between inbox processed_at stamps during a backlog drain, so one linkset cost about k × (8 + 0.14 k) ms. Throughput therefore depends on corpus composition. Measured on a seed-like issuer subset, the early CVE range (CVE-1999…CVE-2011) carries 2.6 events per linkset, with single linksets above 1,500 facts; later ranges carry about 1.5.
Linksets per hour are not comparable across CVE ranges, hosts or import phases. The 78,000 per hour a fresh install showed in the early range, while the seed import was still replaying, and the 390,000 per hour an earlier install showed over later ranges after its import had finished, came from the same vulnerabilities-worker image and the same seed. The earlier install’s own early-range rate, during its import, was about 114,000 per hour. The projection host now logs events applied, linkset rebuilds, elapsed time and events per second once per lease-heartbeat window while it drains, so the rate is measured rather than reconstructed from row counts.
With one transaction per batch and one rebuild per distinct linkset, the same early-range composition drains at 156.9–160.7 events/s, against 63.6–77.3 for the per-event projector in interleaved runs on one host (2.1–2.5x). Re-verify with the benchmark in StellaOps.Vulnerabilities.Facts.Tests: STELLAOPS_PROJECTION_BENCH_OUT=<report file> StellaOps.Vulnerabilities.Facts.Tests.exe -method "*ConsensusProjectionThroughputBenchmark*".
The rebuild is set-based per batch (SPRINT_20260929_004 PCD-182). A batch’s distinct linksets are passed to PostgreSQL as arrays: one facts read, one content-guarded linkset upsert whose RETURNING names the rows that moved, and one delete for linksets that lost their last live fact. Before the upsert, a batch with live facts resolves its exact products through a dictionary insert-if-missing and a key-share read. A race with orphan collection can repeat that bounded resolution pair, up to three attempts; the former three-statement count predates this storage. Until HSR-1 a dist delete and upsert pair made the historical rebuild five statements; the measurements below were taken with that earlier representation. The batch’s consensus.changed events are appended in one outbox batch. Inbox admission stays one statement per envelope. On the same corpus, interleaved runs on one host measured 180.8–224.5 events/s set-based against 70.6–77.0 for the per-linkset rebuild (2.3–3.2x per round, 2026-09-30). The per-linkset figures are lower than the 156.9–160.7 measured for PCD-111 on the same corpus; rates move with host load, so only interleaved pairs are compared. Replay still writes zero hot-row versions. The projection still runs in outbox order; no targeted path exists that projects scan-requested product keys first.
Measured from empty (2026-10-02): the seeded v1.0.0 bundle installed on a fresh PostgreSQL volume on an 8-vCPU, 25 GiB Linux host, with the hub images built from source that carries the set-based rebuild and the bundle’s projection batch of 1,000. The seed import wrote 11,179,140 facts between 05:51 and 07:39 UTC, and the projector drained them concurrently. It applied the 11,179,140 events as 6,917,084 linkset rebuilds at 1,133.7 events/s overall (309.8 to 1,837.5 per logged drain, median 1,129.3) and reached lag 0 at 08:35, 56 minutes after the last fact was written. The lag peaked at 3.8M events near the end of the import. These figures come from a different corpus mix, batch size and host than the benchmark’s and are not comparable with it. Evidence: 20261002-pcd182-from-empty-projection (pruned evidence docs/implplan/_evidence/20261002-pcd182-from-empty-projection/README.md at a095e85589).
Key corrections converge on worker start
A fact’s id hashes its vulnerability id, product key and applicability, so a correction to those identity fields cannot update stored facts in place. FactKeyConvergence (src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Facts/Store/FactKeyConvergence.cs) runs every rule in FactKeyConvergenceRules once per estate, under the source’s normalization fence. RunOnceAsync records a completed pass as a checkpoint row (consumer_id vuln.fact-key-convergence#<rule>#<source>, stream vuln.fact-key-convergence) and skips a recorded rule on later starts, because facts that legitimately keep a matching key (a layered Red Hat product names no platform, a document is no longer retained) would otherwise be read, and their documents normalized, at every start. New documents need no convergence: the pump writes them with the current normalizer. A correction that must run again ships as a rule with a new name. A pass:
- It selects the retained raw documents behind live facts whose key matches the rule’s stale-key predicate, and replays them in the pump’s order (observed time, then digest; a document that is no longer retained sorts last). A fact that two revisions of one record both assert therefore keeps the newer revision’s provenance, as normal ingest leaves it (PCD-823).
- For each document, in one transaction, it normalizes the document again with the current normalizer (
NormalizationPump.RenormalizeDocumentAsync, which leaves the source checkpoint alone). It then removes that document’s provenance from the stale-key facts it no longer asserts (a fact the document yields again is never stale, whatever the predicate says) and tombstones any stale fact with no live provenance left. Each tombstone emits avuln.fact.tombstonedenvelope, so the projector removes or shrinks the stale linkset and emitsconsensus.changed. The envelopes of one document are appended as one batch with one sequence allocation (SPRINT_20261007_063 HSR-14): one allocation per retired fact updated the stream’s singlestream_staterow once per fact inside the document transaction, at a cost that grew with about the square of the count. The re-key is recorded infacts.fact_identity_map: one row per superseded fact and successor (the facts the document now asserts under an origin the superseded fact lost, narrowed to the closest: the same applicability under the superseded key plus scope qualifiers (HSR-16), else the same product key and vulnerability, else the same product key, else the same vulnerability; a NULL successor when there is none), at the tombstone instant. It writes no tombstone revision and no first revision for a successor (SPRINT_20261007_063 HSR-10); an inserted fact that is no successor keeps its first revision. The re-normalization never moves a provenance origin from a later retained revision of the same record back to the replayed document (PCD-826:FactWriter.WriteBatchAsyncwithkeepNewerProvenance, set byRenormalizeDocumentAsync), so a superseded revision the rule selects without its newer revision leaves the newer revision’s provenance in place. The identity mapping reads the document’s assertions fromfacts.fact_provenanceby the document’s digest, so a successor whose provenance stays on a later revision is not among them: the superseded fact is then mapped to the closest of the facts whose provenance names the document, or to a NULL successor (recorded in SPRINT_20261005_002, PCD-826). The fact content still follows the replayed document, as before. The same transaction writes the document’s advisory metadata candidates (facts.advisory_metadata_input), but never over a row that holds a later revision of the same source record in the pump’s order (observed time, then digest): a pass can select a superseded revision whose newer revision it does not replay, because the newer revision’s facts no longer match the predicate (PCD-823). Every other metadata writer applies the same condition (PCD-826, “Metadata selection” below). A candidate under an identity the rule changes (osv-ubuntu-cve-identity) is inserted, as routine ingest would insert it. - A stale fact whose document is no longer retained is left in place and counted, because retiring it would drop a statement nothing can write again.
- A statement identity that the current normalizer refuses (SPRINT_20261007_063 HSR-17, see “Intake refuses per identity” above) is refused per identity in the replay as in the pump. A document-qualified rule replays a marked document whose normalization refuses an identity even when nothing else in it needs the correction, so the refusal is seen. The identity is not written, and the facts an older normalizer stored for it stay as stored: the retirement step skips every stale fact of the refused vulnerability under the refused product key, with or without scope qualifiers, because an older normalizer may have stored the statement unscoped and an unreadable bound gives it no fact id. The document’s other facts converge in the same transaction. The worker logs a warning with the rule, source, vulnerability, product key, document digest and reason, and the summary counts refused identities. A retained document the current normalizer cannot read at all (the qualifier throws) writes nothing, keeps its stored facts, and is counted as unreadable, with a warning naming the document digest and reason (HSR-15). Either way the pass continues with the next document and records its completion. Before HSR-15 one such document stopped the rule at every start; HSR-15 refused the whole document, so its other facts never converged.
The tombstones a pass writes are superseded identities (data identity standard D9), not aged facts. SupersededIdentityReclaimer (src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Facts/Store/SupersededIdentityReclaimer.cs) recognises one by its identity map row (or, before migration 023, its tombstone revision) and deletes it, with any provenance, binary index and identity map rows that reference it, once the tombstone is older than the outbox retention window and than every corpus publication the hub retains. It runs as a leased worker pass of bounded batches, each fenced by the lease and recorded in its pass receipt; it never touches mirror replacements or other tombstones. SupersededRevisionPruner (lease vuln.superseded-revision-prune) then deletes the facts.fact_revision rows of facts that no longer exist, in leased bounded batches. The rules, their evidence and the measured backlog are in retention and custody and revision history.
Convergence writes facts, identity map rows and tombstone events, so it pauses with the worker’s capacity writer gate (SPRINT_20261007_062 DIP-28; see Doctor checks below): a closed gate holds a rule before its fence and its stale-set read, and stops a running rule at a committed document, at most 30 seconds after the gate closes. A stopped rule records no completion; the documents it committed stay converged and the rest still match the rule. The worker runs that rule again every CapacityRecheckInterval (15 minutes by default) until the gate reopens, then continues with the later rules, so a pause never waits for a worker restart.
The gate reads the newest capacity receipt, which the probe writes about once a day, so it cannot see a rule grow the hub within the day; one rule can add tens of gigabytes between two receipts. Each time the gate is open at one of those reads, convergence also measures the live database size (pg_database_size, a few milliseconds) and holds the rule in the same way while that size is at or above the newest receipt’s stop line less CapacityReserveBytes (10 GB by default; see Configuration). Operators set it with the environment variable Vulnerabilities__FactKeyConvergence__CapacityReserveBytes on vulnerabilities-worker (bytes; read at start, so a change needs a worker restart). The stop line comes from the receipt, not from configuration. The reserve leaves ingest its headroom: without it convergence could run up to the stop line, and the receipt that records it would then close the ingest gate too. An operator holds convergence entirely with a reserve larger than the stop line, and Enabled=false still switches it off. The worker logs one line when the reserve starts holding convergence (live size, stop line and reserve) and one when it clears.
A pass costs one read per rule: a scan of that issuer’s facts, 116 s for Red Hat CSAF’s 8.3M rows on a 23.4M-fact hub, run in the background after start. The read (1800 s), each per-document statement (300 s) and every statement of the document write it calls (FactWriter and the metadata-input writer, 300 s each) set explicit command timeouts, because the 30 s driver default is shorter than the read and than a large document’s write. The shared outbox append uses the host-supplied statement budget: every fact-writing entry point configures EventingReliabilityOptions.OutboxCommandTimeoutSeconds to the document writer 300 s budget. Both single and batch append apply it to stream-state ensure, sequence allocation and insert. The ensure can wait on another document transaction as allocation can; a 30 s default timed out during Red Hat CSAF normalization on the loaded local hub (PCD-337). A supplied budget must be positive; hosts that supply none retain their connection command default. The identity mapping reads each retired and asserted fact once, by id, whatever the number of origins it carries: a SUSE CSAF document gives a fact one origin per product and architecture (278,973 origins on 17,792 facts in one pending document, SPRINT_20261007_063 HSR-14). A start that finds the source’s normalization fence held by an ingest cycle waits and retries. Vulnerabilities:FactKeyConvergence:Enabled=false holds a correction back, for example during an incident. The current rules are csaf-qualifier-escapes (Red Hat and SUSE CSAF) and distro-platform (OSV, the Debian security tracker and Alpine secdb: deb and apk facts without an applicability platform). distro-platform also re-keys OSV’s older keys that kept qualifiers, such as Ubuntu’s ?arch=source&distro=<series>, because those carry no platform either. Its first run on a hub that already holds OSV Ubuntu data is the expensive one: about 266k OSV documents normalized again for 3.1M facts on the local seeded hub (plus 332k Debian tracker and 45k Alpine facts), all under the OSV normalization fence, so OSV ingest normalization waits until the rule finishes. A mirror-seeded install converges when it imports a seed exported from a converged hub, because the mirror normalizer carries keys and platforms through unchanged. osv-ubuntu-deb-scheme (OSV) moves Ubuntu ranges stored under generic-exact onto deb: the OSV normalizer maps Ubuntu ecosystems to the dpkg comparator as it maps Debian, where generic-exact made the Findings matcher refuse every Ubuntu range that has a fix (invalid_bounds). It re-normalizes the OSV Ubuntu documents once, under the same fence and at the same order of cost as the first distro-platform run. osv-packagist-scheme (OSV, PCD-823) moves every OSV Composer fact (pkg:composer/...) still stored under generic-exact onto the ordered scheme the normalizer writes for every Packagist ecosystem since PCD-332 (semver). Every OSV fact comes from range events or is product-level (the normalizer does not read OSV versions), so such a row is a range under the wrong comparator, and the Findings matcher refuses one with a fix bound (invalid_bounds, PCD-822). A retained document qualifies only when the current normalizer gives it an ordered Composer fact: a Composer entry without an ecosystem stays generic-exact, and its fact is reasserted, not retired. The replay keeps the bounds, restores a legacy last_affected bound that an older normalizer stored as a fix, and turns an introduced-only range into an open ordered range. The rule runs before osv-ubuntu-cve-identity (disjoint facts) and may overlap osv-drupal-packagist-scheme and osv-last-affected: all three replay through the same normalizer and retire only facts the replay does not assert, so either order ends in the same facts. Measured read-only on the local hub (2026-10-11): 14,043 live facts on 1,559 keys behind 7,577 documents, all retained; 997 of those documents have a newer retained revision of the same record. The replay writes metadata candidates for 7,187 source records: 6,690 rows already hold the replayed revision with the same content, and 497 Drupal contrib rows hold a newer revision that the pass does not replay and keep it. edition-platform-identity (CVE and NVD, SPRINT_20261007_063 HSR-16) re-keys the collapsed facts of documents whose statements now carry scoped identities (see “Scoped retained identity” above). Such a collision needs an inclusive end beside an exclusive one of the same value (CVE) or an exclusive start or inclusive end (NVD), so the qualifier normalizes only documents with that marker, and a document qualifies when the current normalizer scopes one of its facts or refuses one of its identities (HSR-17). Its candidates are the unscoped collapsed shapes cve-less-than-or-equal and nvd-endpoint-inclusion select; the replay asserts the scoped facts and retires the collapsed fact, mapped to its scoped successors. The rules run after those two: where they already replayed a document with this normalizer, they re-keyed it themselves and this pass re-asserts the same facts. Measured read-only on the hub (2026-10-12): 41 documents change (40 NVD, CVE-2022-2675 from cve), 122 facts inserted, 41 retired, 20 revised in place, 114 identity map rows and 183 outbox envelopes; the passes read the documents behind 480,606 NVD and 296,759 CVE candidate facts, and one identity of one NVD document (CVE-2023-48795) stays refused, its stored fact kept. osv-ubuntu-cve-identity re-normalizes retained documents behind OSV Ubuntu facts still keyed by UBUNTU-CVE-.... It writes the confirmed CVE fact, removes the old document provenance and tombstones the old fact only if no other provenance remains. The normal fact events rebuild both issue subjects. Unconfirmed upstream identities remain and are protected by the convergence rule’s reassertion check; the completion record prevents repeated startup work on those rows. csaf-platform (Red Hat and SUSE CSAF) moves versioned rpm facts (Red Hat) and versioned pkg:generic/suse-csaf/ package facts (SUSE) without a platform onto the release their stream names. The subsequent suse-csaf-rpm-keys rule re-normalizes versioned generic SUSE package facts onto the complete RPM keys described above. Unknown releases or missing architecture remain generic and cannot match installed RPMs; reasserted facts are preserved. Existing csaf-platform completion records are retained. Measured read-only on the local hub (2026-10-04, EXPLAIN ANALYZE of the stale-set read): Red Hat 5,908,271 facts behind 23,903 retained documents (8,685,721 provenance rows), read in 170 s; SUSE 3,169,052 facts behind 42,281 documents (18,853,761 provenance rows), read in 508 s; both under the 1800 s read timeout. Layered-product facts (Satellite, OpenShift, JBoss EAP, Ansible) keep no platform and stay matched by the predicate, which the completion record makes a one-time cost. The same pass writes the RPM epoch into Red Hat bounds: CSAF names it in the purl’s epoch qualifier (pkg:rpm/redhat/openssl@1.1.1k-12.el8_9?arch=x86_64&epoch=1), the normalizer writes 1:1.1.1k-12.el8_9, and epoch 0 stays unwritten. Installed versions carry the epoch inline and an RPM comparison reads a missing one as 0, so an epoch-less bound ordered every epoch package above its fix.
7. API surface
All routes are defined in src/Vulnerabilities/StellaOps.Vulnerabilities.WebService/Program.cs and Serving/*.cs. The generated API reference and the committed OpenAPI document list every operation; the canonical baseline is src/Api/StellaOps.Api.OpenApi/vulnerabilities/openapi.yaml.
Route family (under /api/vulnerabilities/v1) | Authorization | Source |
|---|---|---|
readiness, sources | advisory:read | Program.cs |
advisories/{id}, advisories/{id}/symbols, products/{productKey}/issues, issues/{id}/{productKey}/facts, facts/{factId}/evidence | advisory:read | Program.cs |
vex/{id}/{productKey} | vex:read | Program.cs |
exploit-evidence/{id}, products/{productKey}/exploit-evidence | advisory:read | Program.cs |
epss/current, epss/{cveId}/changes, epss/model-date | advisory:read | Program.cs |
corpus/export, corpus/publications/current, corpus/publications/{artifactRef}/manifest, corpus/publications/{artifactRef}/sections/{sectionName} | advisory:read | Program.cs |
events/{stream}, events/{stream}/consumers[/{consumerId}] | advisory:read | Program.cs |
binaries/build-ids/{buildId}, binaries/fingerprints/{id}, binaries/fix-status/{distro}/{release}/{sourcePackage}, binaries/fix-evidence/{factId}/{revisionHash}/{documentDigest}, binaries/patch-coverage[/{id}], binaries/build-id-index | advisory:read | Program.cs, Serving/BuildIdIndexExportEndpoints.cs |
binaries/ops/* | ops:health | Serving/BinaryOpsEndpoints.cs |
symbols/{debugId}, symbols/blobs/{contentHash} | symbols:read | Program.cs |
symbols/sources/*, symbols/marketplace/*, symbols/manifests, symbols/resolve, symbols/status | reads symbols:read; changes also symbols:write and ops.admin | Serving/SymbolManagementEndpoints.cs, Serving/SymbolManifestEndpoints.cs |
issuer submission (POST) | vuln:submit plus a per-issuer client allowlist | Program.cs |
mirror/seeds/import (POST) | advisory:ingest and vex:ingest | Serving/MirrorSeedEndpoints.cs |
maintenance/runs (POST) | ops:health | Program.cs |
The Doctor surface is /doctor/vulnerabilities-web/checks (MapServiceDoctorEndpoints, ops:health). Symbol-management access rules are in symbol-management.md.
Behavior that consumers depend on:
- Readiness is generation-based, never wall-clock.
readyis true once at least one source has a completed generation that ingested a document; a completed generation that saw nothing never declares a corpus ready (GenerationBackedCorpusReadinessProvider). Readiness does not require every source to be healthy; per-source health is thesourcesroute. - Per-source health is computed once, here, over
runtime.source_generation, with the vocabularyhealthy | ingesting | empty | stalled | never-completed. The route is read-only. - Lookup returns global linkset observations: tuple, fact/status/severity summaries and content hash. It is not a tenant decision; a consumer must not invent issuer precedence from the count maps alone.
- Corpus export streams deterministic NDJSON without a BOM, declares its section and format version in
X-StellaOps-Corpus-SectionandX-StellaOps-Corpus-Format-Version, and orders rows withCOLLATE "C"so the digest does not depend on the database locale (HubQueries.StreamCorpusAsync,CorpusMatcherRowSerializer). The order is strict: one row per matcher key; the producer collapses duplicate keys deterministically in SQL. - EPSS is owned by the hub; Scanner and risk consumers read it through these routes.
- Issuer submission accepts DSSE-enveloped OpenVEX from allowlisted internal issuers. The configured
IAsymmetricSignatureVerifierverifies the envelope; the document is fully normalized before the transaction writes anything, and invalid binary subjects return422 issuer_document_invalidwithout partial ingestion. The hub holds public verification material only. While the web host’s capacity writer gate is closed, a verified submission returns503 capacity_writer_pausedwithRetry-Afterand writes nothing (Doctor checks). - Event catch-up (
GET /events/{stream}) lets the control plane consume hub events without openingstellaops_vuln. The response carriesretentionHorizonSeq; below the horizon it adds aretentionblock that points at the retained bootstrap leg (GET /corpus/publications/current). The guard is non-blocking: the page is still served. The shared reliability reader resolves the current stream epoch as a scalar before finding its minimum retained sequence. PostgreSQL can stop at the first(stream, stream_epoch, seq)index entry, rather than scanning the retained stream. Superseded epochs are excluded; an absent or empty current epoch has no horizon. The same reader serves Notify, Integrations, Findings, Scanner and SBOM event feeds (PCD-346). - Cross-database consumer registration (
/events/{stream}/consumers) lets a consumer that commits in its own database report its durable position, which pins the stream against reclamation. The lease that decides when a silent registration stops holding the floor belongs to the hub.
Binary reference derivation
The worker can compare package code with operator-admitted vulnerable/patched reference pairs (Vulnerabilities:Binaries:References, BinaryReferenceWorkflowOptions). Each pair declares a reference ID, CVE, component, ABI, target symbols and exact payload digests. At most 16 pairs and 1 to 256 target symbols per pair are admitted; the default observation limit is 64 binaries per package. Complete exact comparisons can emit affected or fixed. Missing symbols, unknown code, conflicting states or an unknown distro ABI produce no statement, and not_affected is never inferred from absence. The shipped example configuration has an empty Pairs array and is inactive. Signing goes through Signer (signer:sign); the worker then submits the DSSE proof to the issuer route. No private key enters the worker or the hub.
Exact OpenVEX subjects use urn:stellaops:binary:<scheme>:<escaped-variant>:<escaped-value>. Build-id variants are gnu-build-id, pe-cv and macho-uuid. Version-range matcher-rows and the compact vulnerability matcher exclude these identities.
8. The corpus artifact
The exploit-evidence consumer contract distinguishes curated exploitation, exploit availability and probability in Findings, Policy and offline exports.
Consumers that need to match against the corpus (Policy’s gate path, Findings, the CLI’s compact export, offline estates) do not query the hub row by row. They consume a generation-stamped, sectioned artifact with the sections matcher-rows, consensus-inputs, exploit-evidence, reachability-sinks and advisory-metadata, plus the optional compact sections compact-metadata, compact-osv-documents and compact-kev-documents. Each consumer fetches only the sections it declares. The contract (sections, version window, manifest canonicalization, reader) lives in StellaOps.Vulnerabilities.Contracts, so producer and consumer verify identity with the same code.
- Identity. A generation’s identity is its manifest digest. The manifest contains no timestamps, so regenerating the same corpus reproduces the same digest.
- Version window. The legacy section readers implement an n/n-1 window and refuse anything outside it:
consensus-inputsv3/v4,exploit-evidenceandreachability-sinksv1/v2,matcher-rowsv4/v5. The current matcher section carries each fact’s release insideapplicability(platform, omitted when absent), optional exclusiveintroducedExclusiveand inclusivelastAffectedends. Its collapse/order key is (vulnerability, product, platform, scheme, introduced, introducedExclusive, fixed, lastAffected, status), so distinct endpoint/comparison meanings cannot collapse. New meaning is emitted only after the consumers adopt it.advisory-metadatais format v2 only (CorpusAdvisoryMetadata.cs). Thecorpus.generation.completedv2 payload (CorpusGenerationPublication.cs) carries an opaquecorpus-publication:*handle, the manifest digest, the source key and generation, and detached section descriptors. It never carries a path, URL or bucket key. - Single writer. The worker is the only publication writer. Under one lease it takes one repeatable-read input snapshot, streams every required part into the content-addressed root, verifies every digest and length, and then installs the catalogue pointer together with the v2 outbox envelope in one transaction. A failure before that leaves no handle.
- Bounded consensus reads.
consensus-inputsreads 1,000 facts per statement, seeking after(vulnerability_id, product_key, issuer_id, fact_id)through the existing corpus-order index. Text keys use PostgreSQLCcollation and the fact id uses byte order, preserving the existing wire order and bytes. Every page uses the same repeatable-read or serializable transaction; standalone exports own a repeatable-read transaction and caller-supplied weaker isolation is refused. A missing provenance row still refuses the export, including on a later page. The matcher section keeps its separate canonical collapse/order contract. This avoids a full consensus sort before the first row without introducing an unbounded client-side buffer. - Interrupted publication. One progress document under the writer’s control directory holds at most the eight completed section descriptors and at most 65,536 bytes. It pins source/generation, encoding, writer assembly identity and the exact owner input witness. A retry verifies a completed object’s stored digest and length, counts it against the publication budget, and skips its query. Cleanup protects those pending objects as well as catalogue references. Success clears progress. A changed witness or writer invalidates the partial plan, preserving a coherent current snapshot. Source/generation is the completed-generation admission and idempotency key, not a global historical snapshot of every source.
- Input witness. Forward migration
018_corpus_publication_resume_replay_safe.sqlinstalls statement triggers on the seven tables read by the publication queries. Their input writes and transaction marks commit or roll back together. The reader hashes the exact visible transaction-id set and a singleton epoch inside the same repeatable-read snapshot; using only a maximum id would miss an older transaction that commits late. Unrelated transactions do not invalidate progress. After 4,096 trigger statement ordinals, maintenance rotates the epoch and deletes visible marks in one transaction. Forward migration024_corpus_compaction_claim_skip_locked.sqltakes that compaction claim on the singleton withFOR NO KEY UPDATE SKIP LOCKED: a statement that finds the row held by an uncommitted claim leaves compaction to the claimer and does not wait, so writers never queue behind a claim or deadlock with its holder on the outbox stream row, and a rolled-back claim is retried by the next statement. In-flight transactions can add their marks later and remain observable. The singleton epoch is authoritative resume identity; the transaction marks are a bounded optimization cache. The cutoff predicate retains any visible marks newer than the compaction boundary. Neither table owns corpus facts. No initialization or fixture rows are shipped in the migration. PCD-422 freezes the applied 017 SQL unchanged outside active discovery and supplies 018 as its replay-safe successor. Relation-scoped DROP IF EXISTS/CREATE pairs install one correct trigger per input table. Existing owner data, epoch, sequence and mutation marks remain intact, and the historical 017 ledger row is retained without a checksum rewrite. Both migration runners validate discovered resources only; archived 017 is no longer an active startup checksum input. Fresh hosts select 001-016 plus 018, and an existing 001-017 database adds 018. The ordinary startup ledger skips 018 on subsequent starts, while its SQL also tolerates direct replay. Native constrained-owner controls cover both paths, seven trigger witnesses and rollback; the frozen 017 duplicate-trigger failure remains a negative control. The PCD-422 correction is verified against source commit7c29b19684(2026-10-07) through the final 25-case persistence suite and 22-case MirrorRoundTrip class; the migration acceptance receipt (pruned evidencedocs/implplan/_evidence/20261007-pcd422-acceptance.mdat7c29b19684) records source digests, constrained-owner replay, compiled-resource and cleanup boundaries. The resume/coalescing slice is verified againsta1e811954a(2026-10-05 UTC). Re-verify withpwsh tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.Ingestion.Tests/StellaOps.Vulnerabilities.Ingestion.Tests.csproj -Class '*MirrorRoundTripTests*' -BuildProjectReferences -Restore. - Row order and provenance. Both fact sections stream live facts in
COLLATE "C"order, so the section digests do not depend on the server locale.consensus-inputsis ordered by vulnerability, product, issuer and fact id, which the partial indexidx_facts_fact_current_corpus_order(migration014_fact_current_corpus_order_index.sql) serves without sorting the fact table. Every live fact carries provenance: the fact writer adds it in the fact’s transaction, and a path that deletes provenance tombstones a fact it leaves without any. Theconsensus-inputsstream checks each row and refuses the whole publication on a live fact without provenance, instead of filtering that fact out. - Cadence. At most one publication per ingest cycle:
IngestSchedulerrecords candidates per completed source and publishes once after every enabled source has been attempted. A failed publication is reported under the reserved keycorpus-publication, not charged to a source. The worker’s publication store queues its local publication callers with caller cancellation. Waiting for an existing mirror-seed or ingest-cycle publication does not consume the file-lease acquisition budget. Raw retention uses non-queuing local admission and defers while that owner is busy. Both paths still require the exclusive filesystem lease; its cross-process acquisition remains bounded byVulnerabilities:CorpusPublication:LeaseAcquireTimeout(default 30 seconds, positive and at most five minutes). Local admission and the file lease last for the write session and are released on disposal or acquisition failure. A genuine I/O failure remains a failed publication, with no readable handle installed. Operator seeds additionally wait forVulnerabilities:CorpusPublication:SeedPublicationQuietPeriod(default one minute, positive and at most one hour) since the latest completed seed. Rapid pending changes coalesce into that latest generation. The source-operation fence still excludes an import that is currently applying. - Metadata selection. Normalization writes metadata candidates with exact raw-document provenance in the fact transaction, one row per vulnerability, source and source record. Revision rule (PCD-826): a row holds the newest retained revision of its source record in the pump’s order (
COALESCE(last_observed_at, fetched_at), then digest). Every writer replaces a row only from the revision it holds or a later one (CorpusAdvisoryMetadataInputWriter.WriteAsync,keepNewerRevision, which has no default): the routine pump, a fact-key convergence replay (PCD-823) and the backfill, whatever order each writes in. The routine pump already writes in that order, so the condition never refuses it a write; the backfill walks retrieval time, which a reactivated revision’s observation time overtakes. The one exception is thestellaops-mirrorsource (IsLastWriteWinsSource): a mirror seed re-applies a re-delivered document on purpose (NormalizationPump.ReplayMirrorDocumentAsync, an authorized value reversion), and that document can be an older retained revision than the one the row holds, so its routine normalization and re-application stay last-write-wins and its rows are excluded from the repair below.metadata-selection/v1uses only candidates still backed by a live, non-tombstoned fact of the same vulnerability with the row’s document digest; ingestion time never participates. The worker-hosted backfill (Vulnerabilities:CorpusPublication:MetadataBackfill, default off) derives candidates for documents normalized before the projection existed. - Metadata revision repair (PCD-826). Rows written before the revision rule may hold an older revision. The worker-hosted
CorpusAdvisoryMetadataRevisionRepair(Vulnerabilities:CorpusPublication:MetadataRevisionRepair, default off) rewrites each such row from the newest later retained revision (same source key and upstream reference, at or before the source’s normalization checkpoint) that yields the same row under the current normalizers, through the same writer and condition. It rewrites a row only when a live fact of the row’s vulnerability carries that revision’s digest, because otherwise the rewritten row would leavemetadata-selection/v1; such rows are counted as held by fact provenance and stay as they are. It never inserts a row. The candidate read and each batch (whole source records,BatchSizedefault 500 rows) need an open capacity writer gate; each batch holds the source’s normalization fence and commits on its own. A closed gate or a held fence ends the attempt, and the worker retries afterRetryInterval(default 15 minutes) from the rows that still hold an older revision. A later revision that the current normalizers refuse (for example, a payload without a required member) leaves its record’s rows as they are, and the batch continues. A completed source is recorded on the checkpoint streamvuln.advisory-metadata-revision-repair(consumer_idvuln.advisory-metadata-revision-repair#<pass>#<source>) and skipped on later starts. Passv1ran before the fact-provenance repair below; passv2(the current one) reads nothing for a source until that repair has recorded the source’s completion, because the rows held by fact provenance are the ones it unblocks. While the provenance repair is enabled for the source the worker retries afterRetryInterval; otherwise it leaves the source until the next start. - Fact-provenance revision repair (PCD-826, owner decision 2026-10-12). A provenance origin kept the digest of the first document that asserted it before the VULN-B7 ruling (2026-08-26), and a fact-key convergence replay of a superseded revision moved an origin back before PCD-826, so a provenance row can name an older retained revision of its source record than the newest revision that asserts the same fact. The worker-hosted
CorpusFactProvenanceRevisionRepair(Vulnerabilities:CorpusPublication:ProvenanceRevisionRepair, default off) moves each provenance row of a live fact held by a superseded retained revision (a later retained revision of the same source key and upstream reference exists at or before the source’s normalization checkpoint) to the newest later revision whose current normalization asserts the same fact id under the same origin kind and reference with the stored revision hash. It updates the row’s document digest, verification status and signing-key fingerprint (as the fact writer does for a re-assertion) and writes no fact, revision, outbox or metadata row. A row no later revision asserts (a dropped or re-keyed fact), a row whose newest asserting revision asserts other content, and a row behind a revision the current normalizer refuses stay as they are and are counted. It never moves a row to an older revision. The candidate read (superseded revisions with their provenance row count) and each batch (whole revisions,BatchSizedefault 5,000 provenance rows; one revision with more rows forms a batch of its own) need an open capacity writer gate; each batch holds the source’s normalization fence and commits on its own. Thestellaops-mirrorsource is excluded, as for the metadata repair. A completed source is recorded on the checkpoint streamvuln.fact-provenance-revision-repair(consumer_idvuln.fact-provenance-revision-repair#v1#<source>) and skipped on later starts; a resumed pass reads the rows still on a superseded revision again. - Retention.
RetainedCorpusPublicationStorekeeps the newestRetainedPublicationCountpublications (default 2) withinMaxPublicationBytes(default 16 GiB) andMaxRetainedBytes(default 32 GiB). A pruned handle becomes a tombstone forTombstoneRetention(default 14 days) and its routes answer410 corpus_publication_reclaimed; an unknown handle answers 404.GET /corpus/publications/currentpublishes anartifactRetentionblock whoseretainedArtifactRefsmembership is the consumer’s horizon test. Producer deletion never depends on consumer state. - Stored encoding. The writer stores every section gzip-encoded (
Vulnerabilities:CorpusPublication:SectionEncoding, defaultgzip; the closed set isidentityandgzip). One streaming pass feeds the canonical NDJSON into SHA-256 and length accounting, which stay the section’s identity in the manifest, and into the encoder, whose output is hashed and counted as the stored object and committed under its own digest. The manifest is unchanged by the encoding, so a generation’s identity never depends on a compressor’s output. The catalogue (runtime.corpus_publication_section), thepublications/currentresponse and thecorpus.generation.completedv2 descriptor carryencoding,storedDigestandstoredBytesbeside the identitydigestandbytes; the v2 payload writes the three only for an encoded section.MaxSectionBytesandMaxPublicationBytesbound the stored bytes;MaxLineBytesstill bounds one canonical line. Readers verify the stored digest of what they download, then decode and check the decoded bytes against the identity digest, length and final newline before any row is used (CorpusSectionEncodingin the contracts SDK). - Whole and ranged reads through the gateway. Manifest and section responses carry digest-backed
ETag,DigestandX-StellaOps-Content-Digestheaders and support byte ranges withIf-Range. For a section these describe the stored object; the response also declaresX-StellaOps-Corpus-Section-Encodingand the identity inX-StellaOps-Corpus-Section-Digest, and a gzip section is served asapplication/gzip. Whole and ranged section reads are supported through the Router gateway: its exact HTTP route^/api/vulnerabilities/v1/corpus/publications/[^/]+/sections/[^/]+$streams the stored object fromhttp://vulnerabilities.stella-ops.local:8080and passesRange,If-Range,Content-Rangeand the headers above through unchanged. The section file is not carried in a Messaging frame, where a whole large section exceeds the frame’s JSON string limit. The manifest andcurrentroutes remain published endpoints. The operational procedure is in the vulnerability operations runbook. - Unstamped snapshots.
stella vuln-db export --from-hub <url>writes section bytes verbatim plus the canonical manifest, without a generation. Such a snapshot cannot enter a matcher host’s generation-keyed working set. The/corpus/exportroute is not a retained-generation or bootstrap authority.
9. Mirror seeds and the release-bundle seed
The hub exports and imports tenant-free mirror seeds; the procedure is in mirror-seed.md.
- Export is a CLI operation (
stella mirror seed export), projected from one retained publication bysrc/Cli/StellaOps.Cli/Services/MirrorSeedPublicationProjector.cs. - Import is
POST /api/vulnerabilities/v1/mirror/seeds/import(application/vnd.stellaops.mirror-seed+ndjson), staged underVulnerabilities:MirrorSeed:StagingRootand bounded byMaxBundleBytes(default 32 GiB). The worker’sMirrorSeedPublicationWorkerpublishes the imported corpus through the ordinary publisher, including when online ingestion is disabled or the estate is sealed. - Customer database footprint. A from-empty import of a 4,905,152,865-byte seed measured 44,908,713,663 bytes in this owner database on 2026-10-08. The measured source predates HSR-1 and still had the stored distribution table. The per-layer split and its limits are in mirror-seed.md.
- Release bundle. The bundle carries both hub deployables and a signed seed under
data/vulnerabilities/. The installer verifies its checksum and imports it after the services are healthy, and records the outcome instate/vulnerability-seed-import.json(devops/release/bundle/README.md, “Offline vulnerability data”). - Data notices. Release staging (
devops/release/bundle/tools/stage-vulnerability-seed.py) admits only sources cleared in the registerdevops/release/bundle/tools/vulnerability-seed-cleared-sources.json, which points atthird-party-licenses/DATA-NOTICES.md(“Hub mirror seeds”). Cleared:alpine-secdb,cve,debian-security-tracker,kev,nvd,redhat-csaf. Refused by name:osv,suse-csaf(counsel ruling indocs/legal/decisions/mirror-seed-data-notices-counsel-thread-reply.md). - Record bound. On this commit one advisory claim group is bounded by
MirrorBundleStream.MaximumAdvisoryCharacters(16 Mi characters,src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Contracts/Mirror/MirrorBundleStream.cs), and the projector refuses an oversized group instead of splitting it. - In flight at the verified commit (
docs-archive/implplan/SPRINT_20260925_004_Program_day_one_customer_defects.md): DAY-10 makes the projector split an oversized claim group instead of refusing it. DAY-11 adds the per-record OSV upstream rule, theoriginsfield and per-upstream manifest counts, which is the condition for clearingosv. Neither is on this commit. - Update 2026-09-27 (not a re-verification of this page): both have landed. DAY-10 (
8a942575b1): the CLI projector writes an oversized advisory as consecutive continued records (src/Cli/StellaOps.Cli/Services/MirrorSeedPublicationProjector.cs). DAY-11 (c9bf063c47):osvis cleared per record throughosvUpstreamsin the register above, so the record-bound and data-notices bullets describe the verified commit, not the current tree.
10. Configuration
| Section | Purpose | Source |
|---|---|---|
Vulnerabilities:Ingestion with :Scheduler (Enabled, EnabledSources), :Throttling, :SourceRetry, :MirrorImport, :Sources:* | connector scheduling, pacing, retry and per-source settings such as Sources:Nvd:ApiKey and Sources:DistroBinaryCorpus | StellaOps.Vulnerabilities.Ingestion |
Vulnerabilities:Authority (Authority, ClientId, ClientSecret, TokenCacheDirectory) | the worker’s client-credentials identity | StellaOps.Vulnerabilities.Worker |
Catalog:Replication:EnvironmentState | installation posture replica read before each online fetch | VulnerabilitiesWorkerPostureRegistration.cs |
Vulnerabilities:Projection (BatchSize, PollInterval) | consensus projector pacing; BatchSize (default 200) is also the size of one destination transaction | StellaOps.Vulnerabilities.Facts |
Vulnerabilities:FactKeyConvergence (Enabled default true, BusyRetryInterval default 1 min, BusyRetryLimit default 60, CapacityRecheckInterval default 15 min, validated positive at startup, CapacityReserveBytes default 10,000,000,000, validated non-negative; environment variable Vulnerabilities__FactKeyConvergence__CapacityReserveBytes) | the worker-start convergence of stored facts onto corrected normalizer keys; a rule the capacity writer gate or the live-size reserve stopped runs again after CapacityRecheckInterval; convergence waits while the live database size is within CapacityReserveBytes of the newest receipt’s stop line | StellaOps.Vulnerabilities.Facts |
Vulnerabilities:SupersededIdentityReclaim (Enabled default true, Interval default 30 min, BatchSize default 1000, MaxBatchesPerPass default 200, LeaseTtl default 2 min) | the leased reclamation of key-convergence tombstones; validated at startup | StellaOps.Vulnerabilities.Facts |
Vulnerabilities:SupersededRevisionPrune (Enabled default true, Interval default 30 min, BatchSize default 10000, MaxBatchesPerPass default 200, LeaseTtl default 2 min) | the leased prune of the facts.fact_revision rows of facts that no longer exist; validated at startup | StellaOps.Vulnerabilities.Facts |
Vulnerabilities:CorpusPublication (Enabled, RootPath, byte limits, RetainedPublicationCount, TombstoneRetention, MetadataBackfill with Enabled, BatchSize, Sources and CapacityRecheckInterval default 15 min, MetadataRevisionRepair (Enabled default false, BatchSize default 500, RetryInterval default 15 min, Sources default every repairable source; validated at startup), ProvenanceRevisionRepair (Enabled default false, BatchSize default 5,000 provenance rows, at most 100,000, RetryInterval default 15 min, Sources default every repairable source; validated at startup)) | retained publication; the metadata backfill (a pass the capacity writer gate stopped runs again after CapacityRecheckInterval), the metadata revision repair and the fact-provenance revision repair | RetainedCorpusPublicationStore.cs, CorpusAdvisoryMetadataBackfill.cs, CorpusAdvisoryMetadataRevisionRepair.cs, CorpusFactProvenanceRevisionRepair.cs |
Vulnerabilities:MirrorSeed (StagingRoot, MaxBundleBytes) | seed import | MirrorSeedOperations.cs |
Vulnerabilities:Binaries:Analysis, :References, :FootprintBudgetBytes | binary corpus analysis and reference derivation | StellaOps.Vulnerabilities.BinaryAnalysis, .Binaries |
Vulnerabilities:SymbolBlobStorage, Vulnerabilities:FingerprintBlobStorage | blob storage roots | StellaOps.Vulnerabilities.Symbols, .Fingerprints |
Doctor:Registration | opt-in registration with the Platform Doctor registry; disabled by default in the bundle | Program.cs |
Installation posture for online ingestion
The worker reads installation posture from its own local catalog replica before each online HTTP attempt, for every online source, binary corpus index, EPSS and package payload retrieval. A declared seal strengthens the local egress policy; a declared unseal cannot weaken a local seal or its destination rules. Missing or unreadable replica state refuses the attempt, and a witnessed absence uses local bootstrap configuration. Local mirror-file import is unaffected. The connector’s bounded redirect handler rechecks each destination, refuses HTTPS downgrades and strips credentials across origins. A local policy refusal surfaces once as AirGapEgressBlockedException. The drain requires an identity granted catalog:replicate (VulnerabilitiesWorkerPostureRegistration.cs).
11. Doctor checks
vulnerabilities-web hosts its own checks (AddServiceDoctorChecks("vulnerabilities-web")) in addition to the shared service checks:
| Check id | Class |
|---|---|
doctor.vulnerabilities.retention.fact-revision-runway | FactRevisionRetentionRunwayCheck (WebService/Retention/FactRevisionPartitionRunwayService.cs) |
doctor.vulnerabilities.binaries.table-census | BinaryPlaneTableCensusCheck (StellaOps.Vulnerabilities.Binaries/Operations/BinaryPlaneDoctorChecks.cs) |
doctor.vulnerabilities.binaries.corpus-footprint | BinaryCorpusFootprintCheck (same file) |
doctor.vulnerabilities.capacity.budget | HubCapacityBudgetCheck (WebService/Retention/HubCapacityBudgetCheck.cs) |
doctor.vulnerabilities.capacity.budget reads the newest runtime.capacity_probe_receipt, which the web host’s leased capacity probe writes at startup and then daily. The probe measures stellaops_vuln and every registered Vulnerabilities table against the data-identity register (migration 021), and the event rate of eventing.outbox and eventing.inbox (logical events written in the trailing 7 days / 7) against the database’s event-rate budget. It also counts two duplication lines (SPRINT_20261007_062 DIP-14, Facts/Store/HubDuplicationLines.cs): ingest.raw_document rows beyond one per upstream identity (an index-only scan of ix_raw_document_upstream_revision, budget 310,000) and key-convergence tombstones in facts.fact_current past the superseded-identity reclaimer’s cutoff (over idx_facts_fact_current_tombstoned, capped at the stop line, budget 200,000, one reclaim pass). The check fails when the newest receipt is absent, older than 36 hours or at a stop line, and warns at an alarm line. An event-rate or duplication line never pauses a writer. The worker has one capacity writer gate (ADR-039 P7), closed while that receipt shows the database at its stop line (reason capacity-pause:), while it is older than Capacity:WriterGate:StaleAfter (48 hours by default), and while no receipt exists once Capacity:WriterGate:UnmeasuredGrace (20 minutes by default) has passed since the gate started (reason capacity unmeasured:; SPRINT_20261007_062 DIP-13). A worker deployed before the web host that runs the probe therefore pauses after the grace, with a reason that names that deploy. The gate logs one line when it closes or opens, naming hub writers, not one per cycle or batch. While it is closed, every background writer of the worker waits (SPRINT_20261007_062 DIP-28):
| Writer | Where it reads the gate | While the gate is closed |
|---|---|---|
IngestScheduler | at the start of each ingest cycle (which ends with the cycle’s corpus publication) and each binary corpus analysis pass | the cycle or pass is skipped; ingest resumes on the first cycle after a fresh receipt below the stop line |
FactKeyConvergence (FactKeyConvergenceWorker) | before each rule, and between committed documents at most every 30 seconds; an open gate is followed by the live-size reserve check | the rule stops at a committed document and records no completion; the worker runs it again every Vulnerabilities:FactKeyConvergence:CapacityRecheckInterval (15 minutes by default), so convergence resumes without a restart once the gate reopens, and later rules wait for it |
ConsensusProjectionHost | before each drain round, and on the lease heartbeat inside a long drain | the drain stops at the next batch; the checkpoint holds and the lease is kept. No event is lost: the outbox pruner deletes vuln.facts rows only at or below the projector’s checkpoint, whatever their age |
CorpusAdvisoryMetadataBackfill (AdvisoryMetadataBackfillWorker, when enabled) | before each batch | the pass stops at a committed batch; the worker runs it again every Vulnerabilities:CorpusPublication:MetadataBackfill:CapacityRecheckInterval (15 minutes by default) and it resumes from its checkpoint |
MirrorSeedPublicationService | when a seed publication is due | the request stays pending; the next 30-second poll after the gate reopens publishes it |
CorpusAdvisoryMetadataRevisionRepair (AdvisoryMetadataRevisionRepairWorker, when enabled) | before its candidate read and before each batch | the pass stops at a committed batch and the worker retries it after RetryInterval (15 minutes by default) |
CorpusFactProvenanceRevisionRepair (FactProvenanceRevisionRepairWorker, when enabled) | before its candidate read and before each batch | the pass stops at a committed batch and the worker retries it after RetryInterval (15 minutes by default) |
BinaryReferencePreparationWorker (when reference pairs are configured) | before the preparation | preparation waits and re-reads the gate every 5 minutes |
The reclaimers never read the gate, because they are how the database returns below its stop line: RawDocumentRetentionWorker, SupersededIdentityReclaimWorker, SupersededRevisionPruneWorker and the outbox and inbox retention pruners keep running. The rules are in the retention inventory.
The web host has its own capacity writer gate over the same receipts, with the same three causes and settings, and its transition log names hub request writers (SPRINT_20261007_062 DIP-31, owner ruling Q-52). It holds the two request-driven writers, which refuse instead of waiting:
| Writer | Where it reads the gate | While the gate is closed |
|---|---|---|
Operator mirror-seed import (MirrorSeedOperations, POST mirror/seeds/import) | once per import, after the header and argument checks (a permanent 400 answers first) and before the upload is staged; a validateOnly=true request does not read it | 503 problem details titled capacity_writer_paused, the gate’s reason as detail, and Retry-After: 900; nothing is staged or written. Once the gate opens, a 409 baseline mismatch or active source operation can still answer the same import, because those checks need the source fence and the stored cursor. The gate never stops an admitted import midway, so it never leaves a seed partly applied |
First-party issuer submission (IssuerSubmissionService, POST issuers/{issuerId}/vex) | once per submission, after the allowlist, envelope, signature and document checks and before the write transaction | the same 503 response; nothing is written. A permanent rejection (403, 400, 422) answers first, so a 503 means the same request is accepted once the gate reopens |
Reads, Doctor checks, database maintenance runs, the capacity probe and the retention pruners in the web host never read the gate. Retry-After is 15 minutes because only a newer receipt reopens the gate; the probe writes one at web host start, daily, and 15 minutes after a failed pass.
The worker exposes ASP.NET health checks, including owner_database (VulnerabilitiesWorkerDatabaseHealthCheck).
12. Events
| Stream | Producer | Known consumers |
|---|---|---|
vuln.facts | FactWriter | the hub’s own ConsensusProjector; JobEngine’s advisory trigger (one consumer per tenant, jobengine.advisory-trigger-v1.tenant:*) and Scanner’s surface feed (scanner.vuln-surface-feed, opt-in), both over the catch-up route and checkpointed in their own databases |
vuln.consensus | ConsensusProjector | Policy’s VulnGateProjector, over HttpHubEventFeed |
vuln.corpus | the corpus publisher (corpus.generation.completed v2) | the Findings advisory-corpus projection |
Outbox and inbox retention use the hub’s declaration (HubOutboxRetention, HubInboxRetention) in every host that embeds the eventing chain. All three streams are RemotelyConsumed: a consumer in another database registers and reports its durable position through POST /api/vulnerabilities/v1/events/{stream}/consumers/{consumerId} (body streamEpoch, seq; advisory:read), and a pass keeps every envelope above the lowest active registration. On vuln.facts the ConsensusProjector checkpoint is also required, so the stream is never bounded by the window alone; JobEngine’s trigger reports after each pass and Scanner’s feed after each poll. A registration holds the floor only while it is renewed within the 24-hour RemoteConsumerLease; a lapsed registration is logged by name on every pass and holds nothing, so a stopped consumer cannot pin the outbox. DELETE .../consumers/{consumerId} releases a registration at once.
The hub’s outbox prune budget is 5,000 logical events per batch and 40 batches per stream per pass, with a 60 s time budget. While a pass ends on its budget, the next pass starts after 5 minutes instead of 15, which caps one host’s allowance at about 2.4M events per stream per hour. The mechanism and the measurements are in the retention inventory.
An envelope is addressed by its logical position (stream, stream_epoch, seq); the inbox still dedupes on (consumer_id, stream, stream_epoch, seq). Vulnerabilities stores complete outbox envelopes in gzip batches with physical primary key (stream, stream_epoch, first_seq), an endpoint index for catch-up and retention, and a covering time/count index for event rates. Event storage defines the format, startup conversion and exact logical splits. event_id is a UUIDv7 that OutboxWriter mints for each appended envelope, so its uniqueness rests on the producer. In the neutral row layout, Eventing migration 006_eventing_drop_outbox_event_id_index.sql (SPRINT_20261007_063 HSR-18) removed its unused unique index, which measured 4.28 GB with 0 scans on this hub on 2026-10-12.
13. Boundaries and connectors
Other services may source-reference only the classified closed seams Contracts, Consensus, VulnMatch.Core, BinaryArtifacts and Symbols.Client (src/Vulnerabilities/AGENTS.md; ownership-manifest.json). Everything else is a runtime seam: the owner API, versioned events, or the corpus artifact. Architecture conformance rejects new cross-service implementation edges; the legacy-edge register is shrink-only.
Implemented connectors are in src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Ingestion/Connectors/: alpine-secdb, cve, debian-security-tracker, distro-binary-corpus, kev, nvd, osv, redhat-csaf, suse-csaf (a CSAF index connector defined in RedHatCsafSourceConnector.cs) and stellaops-mirror (mirror import), plus EPSS. Source policy, tiering and removals are in connectors-architecture.md. distro-binary-corpus is opt-in per estate, ships no default repository host and fails closed when unconfigured; bring-up is in the vulnerability operations runbook.
14. Retired predecessors
| Predecessor | Replaced by | Archived dossier |
|---|---|---|
| Concelier (advisory ingestion service with its plugins and connectors) | this hub | docs-archive/modules/concelier/ |
| Feedser (Concelier’s feed-normalization core) | the hub’s ingestion and normalization libraries | docs-archive/modules/feedser/; the 2026-09-26 copy is under retired-20260926/ |
| Excititor (VEX ingestion service and worker) | the hub’s VEX facts and vex routes | docs-archive/modules/excititor/; differing 2026-09-26 copies are under retired-20260926/ |
| VexHub (VEX distribution service) | the vex routes, computed from facts.fact_current | docs-archive/modules/vex-hub/ |
| VexLens (VEX consensus service) | StellaOps.Vulnerabilities.Consensus (global) and Policy (tenant-effective) | docs-archive/modules/vex-lens/ |
The frozen sources are under src/__Obsoleted/Concelier/; nothing there is built or referenced.
15. Related documents
- ADR-039 — database-per-service ownership
- vulnerability-data-plane-v2-design.md, the target design and build record
- Scanner advisory evidence identity
- design-challenges-register.md, the DC-xx decisions cited here
- database-ownership-matrix.md
- Vulnerability operations runbook
Bounded multi-issue fact reads
Source verified against commit de98170965 (2026-10-10). Re-verify from the repository root with:
pwsh tools/scripts/test-targeted-xunit.ps1 -Project src/Vulnerabilities/__Tests/StellaOps.Vulnerabilities.WebService.Tests/StellaOps.Vulnerabilities.WebService.Tests.csproj -Class "*IssueFactsBatchEndpointTests" -BuildProjectReferences
POST /api/vulnerabilities/v1/issues/facts:batch requires advisory:read, matching the point lookup’s authorization. Its JSON request is {"issues":[{"vulnerabilityId":"CVE-2026-0001", "productKey":"pkg:npm/example"}]}. The response contains an issues array in request order; every member carries the exact requested keys and a facts array, including an empty array when no live provenance-backed fact exists. Fact fields and their order within each issue match the point lookup, including inclusive bounds, introduced-exclusive bounds and platform.
The request accepts 1–200 distinct keys, vulnerability identifiers up to 256 characters and product keys up to 4,096 characters. Invalid or duplicate keys return 400. Request and response bodies are bounded to 8 MiB and a response to 20,000 facts. Exceeding a resource bound returns 413 before any partial result is exposed. Clients can reduce the key group; a single issue that exceeds the bound must remain an explicit refusal.
One indexed statement reads the complete key group from one PostgreSQL statement snapshot. Tombstoned and provenance-less facts are excluded, as in the point lookup. Both explicit empty results and input order survive grouping; a database failure is never converted into empty facts. The owning service and closed Contracts SDK define this API. It introduces no cross-service database connection, migration or new transport. Deploy the Hub before enabling its Policy caller.
