Vulnerability Operations Runbook

Audience: Platform on-call and operators triaging vulnerability scanning, advisory-feed, and findings problems in a Stella Ops deployment.

Purpose: Provide a fast, deterministic path to triage, mitigate, and evidence vulnerability-pipeline issues — invalid scanner packs or advisory feeds, unhealthy Scanner / Findings / vulnerabilities-web / vulnerabilities-worker services, and stuck or failed jobs — while preserving reproducibility (hashes and digests) end to end.

Supported Scope

This runbook covers vulnerability operations for Docker Compose, Offline Kit, signed release manifests, and host/service-manager deployments.

Kubernetes and Helm mock overlays are retired. Do not use helm template, helm upgrade, helm rollback, or kubectl scaling/log commands for Stella Ops vulnerability operations.

Current owner: src/Vulnerabilities; the old Concelier/VexLens services and schemas are retired. Related runbooks: VEX Operations · Scanner worker not processing jobs

Triage

  1. Capture the affected tenant, environment, scanner pack, advisory feed bundle, and release manifest digest.
  2. Verify scanner pack and advisory feed hashes against the signed release or Offline Kit manifest.
  3. Check Scanner, Findings, vulnerabilities-web, and vulnerabilities-worker health through supported service endpoints and host logs.
  4. Review queue depth and failed job evidence from the Stella Ops UI/API.

Mitigation

  1. Pause affected vulnerability jobs if the feed or scanner pack is invalid.
  2. Re-import the last known-good Offline Kit feed bundle when feed corruption is confirmed.
  3. Restart only the affected live service (vulnerabilities-web, vulnerabilities-worker, Scanner, or Findings) through Docker Compose or the host service manager.
  4. Re-run the failing vulnerability job and compare output digests.

Evidence

Retained corpus publication bring-up

vulnerabilities-worker publishes at most one retained corpus publication per ingest cycle (IngestScheduler -> CorpusArtifactProducer.PublishCompletedGenerationAsync) into the compose_vulnerabilities-corpus-publications volume; vulnerabilities-web serves it read-only at /api/vulnerabilities/v1/corpus/publications/**. The cadence was one publication per completed source generation until the owner ruling of 2026-08-27 (sprint 003 VULN-B8 (a)); it is now taken once, after every enabled source and the EPSS snapshot have been attempted, and is stamped with the last source in configured EnabledSources order that completed a publishable (non-binary-corpus) generation in that cycle. A cycle in which nothing completed publishes nothing.

The scheduler logs the decision explicitly — grep the worker for either line:

Corpus publication for this cycle: ONE artifact corpus-publication:<hex> stamped '<source>' generation <N> (<k> of <n> enabled sources completed a publishable generation).
No corpus publication for this cycle: none of the <n> enabled sources completed a publishable generation.

A failed publication is a CYCLE outcome, not a source outcome: it appears as Ingest cycle corpus-publication: FAILED (...) and never marks a healthy source failed. Alongside those, watch runtime.corpus_publication / runtime.corpus_publication_current, the volume’s .tmp/ growth, and the worker’s Ingest run for <source> completed generation N / Ingest cycle failed for source lines, which bracket each source attempt.

Two blockers measured on the first live bring-up (2026-08-26) and their fixes:

  1. Volume root owned by root. Both hub images run as uid 10001 and carry no /app/data, so a volume first created by any container mount is root:root 755 and every attempt fails with UnauthorizedAccessException: Access to the path '/app/data/corpus-publications/.control' is denied. Fix once, while the volume is empty: docker run --rm --user 0:0 --entrypoint sh -v compose_vulnerabilities-corpus-publications:/v stellaops/vulnerabilities-worker:dev -c 'chown 10001:10001 /v', then docker restart stellaops-vulnerabilities-worker (the boot cycle re-fires the trigger). Verify from inside the worker: ls -ld /app/data/corpus-publications shows stella stella. Hub images built from 2026-09-27 onward ship /app/data/corpus-publications owned by uid 10001, so Docker creates a fresh volume already owned by the service user and this step is needed only for an existing root-owned volume or an older image.
  2. A stored fact identity string with boundary whitespace (upstream OSV alias "CVE-2020-13665 ") failed the advisory-metadata wire contract (CorpusWireFormatException: Advisory-metadata field 'vulnerabilityId' must be non-empty and have no boundary whitespace) after ~10 GB of the other sections had streamed. VulnFact now trims identity strings on construction and migration 003_fact_identity_representation_convergence.sql converges rows written before that guard at the next hub boot. Detect a recurrence with SELECT count(*) FROM facts.fact_current WHERE vulnerability_id ~ '^\s|\s$' OR product_key ~ '^\s|\s$' (expected 0); never hand-edit hub tables.

Success is GET /api/vulnerabilities/v1/corpus/publications/current = 200 with a manifest digest, one runtime.corpus_publication_current row, and one eventing.outbox row on stream vuln.corpus per publication. Before the first run check free space on the volume host and the database host. Sections are stored gzip-encoded (SPRINT_20261002_004 CSB-1), and MaxSectionBytes (8 GiB), MaxPublicationBytes (16 GiB) and MaxRetainedBytes (32 GiB) bound the stored bytes. Measured 2026-10-03 on the local hub (cve generation 196): 18,468,485,933 B of canonical NDJSON stored as 1,760,101,909 B (10.5x; consensus-inputs 10.87 GB to 1.21 GB, matcher-rows 5.69 GB to 222 MB, advisory-metadata 1.03 GB to 193 MB). Plan the volume for the current and previous stored publications plus one write in flight. Consumers decode a section as they read it, so the plaintext size is what a consumer’s import processes, not what it downloads. The plaintext size outgrew the caps before CSB-1: consensus-inputs passed 8 GiB in late September 2026.

Measured first live publication (2026-08-26, kev generation 45): ~16 min from generation completion to readability; 10,418,775,281 B across five sections (matcher-rows 4.20 GB, consensus-inputs 5.96 GB, exploit-evidence 40.6 MB, advisory-metadata 221 MB, reachability-sinks 0); the store discards the parts of a failed attempt before the next one starts (nothing is reused across attempts), and each completed, pumped source generation publishes again, so a full cycle can stream several publications back to back. docker restart of the worker re-fires the trigger at boot via kev.

Distro binary-corpus bring-up (build-id plane)

Compressed index containers, fixed 2026-09-04 (D-BIN11-8) — point the repository at whichever index your mirror publishes. The stage now strips the transport container off the index before parsing it, and it decides on the BYTES rather than the URL, so an endpoint with no extension, a redirect, or a proxy-renamed file all read correctly. Accepted: plain text, gzip, xz, bzip2, and a tar (compressed or not) whose APKINDEX / Packages member holds the index — which is what APKINDEX.tar.gz is. A tarball holding exactly one file is accepted whatever that file is called. Refused by name, with the reason in the log rather than a silent empty result: .zst, because this build ships no zstd decoder.

Until 2026-09-04 none of that existed and apk-index could not read a single real Alpine mirror (D-BIN11-7): every cycle ended in Repository index kind 'apk-index' enumerated zero packages. If you are running an image built before that date, this is still true, and deb-packages behaves the same way against a Packages.gz/.xz URL — on an older image, use a PLAIN Packages index. Either way the stage fails closed: zero rows written, the checkpoint held behind the document, so nothing needs cleaning up and an upgraded build re-reads the same stored document.

Still unsupported, and this one is a capability gap rather than a container: rpm-repomd. repomd.xml enumerates no packages at all — it only locates primary.xml.gz — so the stage refuses it loudly instead of recording an empty repository as analysed.

Alpine package URLs, fixed 2026-09-05 (D-BIN11-9) — an image built between the two fixes enumerates the index and then 404s every package. The symptom is unmistakable and it is not a mirror problem: the log reads Index …/main/x86_64/APKINDEX.tar.gz enumerates N packages and is then followed by one Skipping <pkg> <ver>: <uri> answered 404. per package, with the architecture DOUBLED in the URI (…/main/x86_64/x86_64/<name>-<version>.apk). binary_identity and binary_package_map stay at zero rows and the footprint does not move. An Alpine repository publishes APKINDEX.tar.gz and the .apk files in the SAME directory, so the package path is resolved beside the index; the stage was re-stating the architecture that the index URL already ends in. Nothing to configure and nothing to clean up — the run writes no rows and holds its checkpoint behind the document, so an upgraded build re-reads the same stored index. Debian is unaffected: a Packages stanza states a repository-root-relative Filename, which is rebased to the segment before /dists/ and always was.

Note for whoever repeats the run: those per-package fetches are rate-limited as of 2026-09-07 (VULN-B12) and were not before. The payload client now sends the hub User-Agent and paces itself per host at a shipped 2.0 rps initial / 0.25 floor / 8.0 ceiling, AIMD, honouring Retry-After and retrying a 429/503 instead of counting the package as skipped. An image built before that date issues one anonymous request per package as fast as it can.

The pacing is per HOST and is shared with the index fetch, so a bounded pass of 8 x 2,000 packages spends on the order of an hour on politeness alone. An estate serving the corpus from its own mirror should raise that host’s cap rather than wait: Vulnerabilities__Ingestion__Throttling__HostOverrides__<host>__InitialRps. Pointing the repository at your own mirror (below) is the standing recommendation for this source anyway.

A truncated index no longer passes as a short one, and an index nobody can vouch for no longer authorizes deletion (VULN-B11, 2026-09-07). A repository index cut off mid-transfer is a well-formed PREFIX: gzip decompresses it without an exception, the parser reads it without an exception, and it enumerates a real but incomplete subset. Nothing below the connector can tell it from a genuinely small repository — and a completed Full-inventory generation is what authorizes D4 absence-tombstones, so that prefix could authorize deleting binaries that still exist upstream.

Declare an anchor per repository and the connector checks the bytes against it:

Vulnerabilities__Ingestion__Sources__DistroBinaryCorpus__Repositories__0__IndexSha256: "<hex>"
Vulnerabilities__Ingestion__Sources__DistroBinaryCorpus__Repositories__0__IndexSizeBytes: "<bytes>"

Either one is enough. IndexSha256 is the stronger of the two: it is the only check that catches a mirror holding a truncated FILE, because that transfer is complete and correctly framed and no transport check can see it. IndexSizeBytes catches truncation but not substitution, and is offered because a directory listing gives a length for free. The transport’s own Content-Length is always checked and needs no configuration.

If you declare neither, nothing breaks — the estate loses absence authority, not ingestion. The run still fetches, still stores the index, and still fills the build-id plane, because build-id facts are additive and true whether or not the enumeration was complete. What changes is that the generation is recorded inventory_scope = 'delta' instead of 'full', and by the connector SPI’s own rule a Delta generation may never authorize an absence-tombstone. The worker says so once per index per run:

Distro binary-corpus index <uri> declares no IndexSha256 or IndexSizeBytes, so a truncated index
cannot be told from a short one. The run is recorded as a Delta inventory and can NOT authorize
absence-tombstones.

An air-gapped or pinned corpus should turn that degradation into a refusal, because there an unanchored index is a misconfiguration rather than a fact of life: Vulnerabilities__Ingestion__Sources__DistroBinaryCorpus__RequireDeclaredIntegrity: "true". It is OFF by default because a LIVE mirror re-publishes its index whenever the repository changes, so a pinned digest would have to be re-declared daily and defaulting to a refusal would break every unpinned estate on upgrade.

Check which you are getting:

SELECT generation, state, inventory_scope, completed_at
FROM runtime.source_generation
WHERE source_key = 'distro-binary-corpus'
ORDER BY generation DESC LIMIT 5;

distro-binary-corpus is the source that fills the hub’s build-id plane (binaries.binary_identity + binaries.binary_package_map). It is opt-in per estate, it is absent from every default source set, and it fails closed when unconfigured — a corpus-less deployment is fully functional and its exported build-id index is simply empty.

Point it at your own mirror, not an upstream CDN

Owner ruling 2026-09-04: this product ships no default host and makes no default outbound call, so no repository URL appears in any tracked file. The recommended target is the mirror the estate already runs, not dl-cdn.alpinelinux.org or any other public CDN. One bounded pass fetches thousands of package archives, which is mirror-shaped traffic; a mirror you pin also gives the same corpus twice, which a CDN moving under you does not. An air-gapped estate serves APKINDEX and the .apk files from its own host and needs no outbound access at all.

The configuration is deliberately split in two:

FileTracked?Carries
devops/compose/docker-compose.vulnerabilities.binary-corpus.override.ymlyesenables the source; repository coordinates (distro/release/architecture/index kind); the capacity lever
devops/compose/docker-compose.vulnerabilities.binary-corpus-repositories.local.override.ymlno (git-ignored; .example template tracked beside it)the repository IndexUri— your mirror

Apply them after docker-compose.vulnerabilities.g2.override.yml, local file last. Compose merges environment maps by key, so EnabledSources__8 extends G2’s 0..7 rather than replacing it.

The one registered exception to the no-tracked-URL rule is the isolated BIN-8 rehearsal compose, docker-compose.binary-fold.scratch.yml. It names its own internal fixtures service, which resolves only inside that rehearsal network. It is never an estate overlay; see docs/modules/vulnerabilities/binary-fold-rehearsal.md.

What “fails closed” means here, precisely

Runtime cost, stated plainly

Enabling this source makes vulnerabilities-worker perform outbound HTTP from the customer’s own deployment, inline in the ingest cycle (IngestScheduler runs the analysis stage synchronously after the source’s generation completes). Package bytes are transient: each payload spools to a FileOptions.DeleteOnClose temp file and cannot outlive its handle, and package archives never become ingest.raw_document rows — that split is the design §9.1 capacity invariant. What is retained is the derived identity: build-id, file digest, format/architecture/type/stripped flag, in-package path, and the package’s purl/version/distro coordinates.

Peak transient disk is capped per package, independently of how many packages a pass analyses: MaxPackageBytes 512 MiB, MaxEntryBytes 64 MiB, MaxUncompressedBytes 1 GiB, MaxEntries 50000.

The capacity lever

Vulnerabilities:Binaries:Analysis:MaxPackagesPerDocument (env Vulnerabilities__Binaries__Analysis__MaxPackagesPerDocument) bounds the packages analysed per index document. Default 250.

It bounds a serial loop — not memory, not document size, not any database column or wire payload. Because per-package disk is capped separately, raising it does not raise peak transient disk; the cost it buys is wall-clock inside the ingest cycle. Never set it to 0 (unbounded) against a Debian Packages index, which enumerates roughly 60k packages.

Reading the result

The worker logs one line per source per cycle; the binary corpus reports corpus counters rather than facts=, because it writes no advisory facts:

Ingest cycle distro-binary-corpus: ok (seen=1, new=1, indexes=1, packages=N (skipped=M), binaries=B, corpusRows=R).

The stage logs its own summary, and logs the truncation explicitly when the lever bites:

Binary corpus analysis for distro-binary-corpus generation <G>: <D> indexes, <N> packages (<M> skipped), <B> binaries, <I> identity and <P> package-map row versions written.
Index <uri> enumerates <T> packages; analysing the first <L> (Vulnerabilities:Binaries:Analysis:MaxPackagesPerDocument).

Measure the plane against design §9.1’s budget with the doctor check on vulnerabilities-web(not the worker, which hosts no doctor checks):

GET /doctor/vulnerabilities-web/checks   ->  checkId "doctor.vulnerabilities.binaries.corpus-footprint"

It requires the ops.health scope (vulnerabilities:health:read policy) and returns footprint.bytes, budget.bytes (Vulnerabilities:Binaries:FootprintBudgetBytes, default 25 GiB = 26843545600) and scope.lever. The check refuses to answer at all unless the measured database actually holds the folded plane, so it can never report “within budget” about the retiring schemas of the same name on the same server.

Advisory-metadata backfill

Symptom: publications succeed but every advisory-metadata row carries null title/severity/ CVSS, and SELECT count(*) FROM facts.advisory_metadata_input is far below SELECT count(DISTINCT vulnerability_id) FROM facts.fact_current WHERE tombstoned_at IS NULL. Measured on 2026-08-26: 0 candidate rows against 669,406 live vulnerabilities.

Cause: migration 002 created the metadata projection after the estate had already normalized its documents, and NormalizationPump writes candidates only for documents past its per-source checkpoint. On a quiet estate (“N seen, 0 new” per cycle) nothing revisits the backlog, so the projection stays empty indefinitely. The standing mechanism is unchanged — the pump writes every NEW document’s candidate in the fact transaction; the backlog needs the one-time pass below.

Procedure (vulnerabilities-worker only; the pass is the hub’s own code path — the same CorpusAdvisoryMetadataInputNormalizer + CorpusAdvisoryMetadataInputWriter the pump uses, over the retained ingest.raw_document payloads, in retrieval order (fetched_at, digest); the writer keeps the newest retained revision in the pump’s order, (COALESCE(last_observed_at, fetched_at), digest), whatever the walk order):

  1. Set VULN_METADATA_BACKFILL_ENABLED=true in the compose .env (optional VULN_METADATA_BACKFILL_BATCH_SIZE, default 200 documents per transaction) and recreate the worker on its recorded compose chain (--force-recreate --no-deps vulnerabilities-worker). Migration 004_advisory_metadata_backfill_checkpoint converges at boot.

  2. Watch docker logs -f stellaops-vulnerabilities-worker | grep 'Advisory-metadata backfill': enabled: sources [...], then a progress line every 5,000 documents (docs/min), then ... for <source> complete: N live documents scanned, M candidate rows written, and finally pass finished ...; idling until the next worker start. Progress and completion per source are also in facts.advisory_metadata_backfill (documents_scanned, candidates_written, completed_at).

  3. Verify coverage: SELECT count(*), count(DISTINCT vulnerability_id) FROM facts.advisory_metadata_input against the live-vulnerability count. Vulnerabilities known only through alpine-secdb / debian-security-tracker have no metadata authority and stay all-null by contract — confirm that is ALL the uncovered set is, rather than assuming it, by grouping the uncovered vulnerabilities’ live facts by issuer_id: only those two issuers may appear. Then check that every candidate is reachable by metadata-selection/v1. Use the selector’s exact predicate — it also requires the live fact to be for the same vulnerability, so a query that joins only on fact_id is a weaker test and will pass rows the selector drops:

    SET max_parallel_workers_per_gather = 0;   -- required, see below
SELECT count(*) FROM facts.advisory_metadata_input i
WHERE NOT EXISTS (
    SELECT 1 FROM facts.fact_current live
    JOIN facts.fact_provenance p ON p.fact_id = live.fact_id
    WHERE live.vulnerability_id = i.vulnerability_id
      AND live.tombstoned_at IS NULL
      AND p.document_digest = i.document_digest);   -- expected 0

    SET max_parallel_workers_per_gather = 0 is not optional. The verification query hits the same wall the walk itself did: the planner turns the anti-join into a parallel hash join over the 33M-row facts.fact_provenance table and the database container’s /dev/shm runs out (53100: could not resize shared memory segment). With parallelism off it is an index-driven plan and completes.

  4. The next completed source generation publishes with the populated section; fetch it by handle from /api/vulnerabilities/v1/corpus/publications/current and spot-check a known CVE. Whole or ranged. A populated advisory-metadata section is ~934 MB. The gateway route ^/api/vulnerabilities/v1/corpus/publications/[^/]+/sections/[^/]+$ proxies section reads over HTTP and streams them, so both a whole GET and Range: bytes=<start>-<end> reads work through stellaops-router-gateway; resume an interrupted read with Range plus If-Range on the ETag. A gateway still running a route table without that row carries the section in one transport frame: measured 2026-08-27, <= 48 MiB ranged -> 206, 64 MiB -> 504 after 30 s, >= 96 MiB -> 502 urn:stellaops:router:invalid-upstream-response, and an unranged read of a large section -> 502. On such a gateway read 16 MiB ranges and concatenate. Accept-Ranges, Content-Range, ETag/If-Range and X-StellaOps-Content-Digest are all correct. Ranges cover the stored object, gzip-encoded since SPRINT_20261002_004 CSB-1 (X-StellaOps-Corpus-Section-Encoding): the concatenated bytes hash to the catalogue’s stored_digest, and their gunzipped bytes hash to its section_digest (X-StellaOps-Corpus-Section-Digest). A section stored before CSB-1 is identity, and its bytes hash to section_digest directly.

  5. Set the variable back to false (or leave it: a re-run scans only the window the pump has since advanced and writes nothing — the writer’s content guard suppresses identical rows).

Scope and cost: only documents whose digest still backs a live, non-tombstoned fact are read (superseded record versions are skipped without detoasting their payload); the pass never advances past a source’s normalization checkpoint. On the 2026-08-26 estate that is ~940k documents and ~25 GB of payload reads (osv 9.6 GB, redhat-csaf 6 GB, suse-csaf 6 GB, cve 1.7 GB, nvd 1.55 GB); expect hours, not minutes, and no additional disk beyond the projection rows. Interrupting the worker is safe: each batch commits with its checkpoint and a restart resumes. To force a full re-walk of one source, DELETE FROM facts.advisory_metadata_backfill WHERE source_key = '<key>' (a checkpoint row, not data — the only hand-run statement this procedure allows).

Measured outcome of the one live pass (2026-08-26 15:08Z -> 2026-08-27 06:23:28Z, verified 2026-08-27 08:30Z; re-verify with the step-3 queries). facts.advisory_metadata_input = 1,247,490 rows over 668,228 distinct vulnerabilities, 1374 MB against 669,579 live vulnerabilities = 99.798%; per source cve 203,383 / nvd 312,274 / osv 349,370 / redhat-csaf 112,455 / suse-csaf 270,008. Unreachable candidates 0. The 1,351 uncovered grouped by issuer_id are exactly debian-security-tracker (1,209) and alpine-secdb (173) — no metadata authority, all-null by contract — so coverage of the derivable population is 668,228 / 668,228 = 100%. The first publication after the pass carried a 979,698,214-byte advisory-metadata section against 221,164,791 B for the all-null one that preceded the backfill. Wall time was dominated by the payload reads, not the writes: cve 1 h 09 m, nvd 1 h 24 m, osv 1 h 32 m, with redhat-csaf/suse-csaf behind them. Two source-data and infrastructure traps hit during that pass are recorded in sprint 003’s Decisions & Risks (invalid UTF-8 in a SUSE CSAF note, handled by treating an unreadable string token as absent; and the /dev/shm parallel-hash wall above).

Advisory-metadata revision repair

Symptom: a corpus publication carries older advisory metadata (title, description, modified) for a record than the newest retained revision of that record. A row of facts.advisory_metadata_input holds a document that has a later retained revision of the same (source_key, upstream_ref), ordered as the pump orders documents (COALESCE(last_observed_at, fetched_at), then digest), at or before the source’s normalization checkpoint.

Cause: before PCD-826 three writers were last-write-wins in an order other than the pump’s: fact-key convergence replays (digest order, until PCD-823), the metadata backfill (retrieval order) and the earliest replays of the same records. Every writer now keeps the newest retained revision (module dossier, “Metadata selection”), so no new drift arises; the rows written before stay until this pass moves them. The stellaops-mirror source is excluded on purpose: a mirror seed re-applies a re-delivered document, which can be an older retained revision.

Measure first (read-only). Per source, the candidate rows and how many of them the pass can rewrite (only a row whose newest yielding revision backs a live fact of the row’s vulnerability; a rewrite of any other row would drop it from metadata-selection/v1):

BEGIN READ ONLY;
SELECT current_setting('transaction_read_only');
SET LOCAL lock_timeout = '3s'; SET LOCAL statement_timeout = '300s';
SET LOCAL max_parallel_workers_per_gather = 0; SET LOCAL jit = off;
WITH rd AS MATERIALIZED (
    SELECT r.digest, r.source_key, r.upstream_ref, COALESCE(r.last_observed_at, r.fetched_at) AS pos
    FROM ingest.raw_document r
    JOIN ingest.normalize_checkpoint c ON c.source_key = r.source_key
    WHERE r.source_key IN ('cve','nvd','osv','redhat-csaf','suse-csaf')
      AND (COALESCE(r.last_observed_at, r.fetched_at), r.digest) <= (c.last_fetched_at, c.last_digest)),
newest AS MATERIALIZED (
    SELECT DISTINCT ON (source_key, upstream_ref) source_key, upstream_ref, digest, pos
    FROM rd WHERE upstream_ref IS NOT NULL ORDER BY source_key, upstream_ref, pos DESC, digest DESC)
SELECT m.source_key, count(*) AS candidate_rows
FROM facts.advisory_metadata_input m
JOIN rd h ON h.digest = m.document_digest
JOIN newest n ON n.source_key = h.source_key AND n.upstream_ref = h.upstream_ref
WHERE (n.pos, n.digest) > (h.pos, h.digest)
GROUP BY 1 ORDER BY 1;
COMMIT;

The count is the pass’s candidate set, not its write count: the pass also needs the newest revision that yields the row (a later revision may name another vulnerability or no record), and that revision must back a live fact of the row’s vulnerability. Size the writes with the selectability query in the PCD-826 receipt before the window.

Procedure (vulnerabilities-worker only, in an authorized hub window; the Worker image must contain PCD-826):

  1. Confirm the newest runtime.capacity_probe_receipt row is fresh and has writer_pause false. The pass checks the capacity writer gate before its candidate read and before every batch and stops at the stop line, so a closed gate makes the pass itself a no-op. That covers the pass only, not the other writers of the same Worker start (step 2).
  2. Precondition for the Worker start: a Worker start also runs every fact-key convergence rule that has not recorded its completion (stream vuln.fact-key-convergence), unless key convergence is disabled or held by its live-size reserve. Decide which before the start: set Vulnerabilities__FactKeyConvergence__Enabled=false in the Worker’s environment (the Worker logs Fact key convergence is DISABLED and runs no rule; stale-key facts stay as stored, and the unrecorded rules run at the next start without the key), or confirm that the reserve (Vulnerabilities__FactKeyConvergence__CapacityReserveBytes, DIP-28) holds them at this database size. On an image without the gate check and reserve (before DIP-28), only the key holds them. The precondition applies to every Worker start in the window, including a restart after a failure and the start that removes the repair key.
  3. Add Vulnerabilities__CorpusPublication__MetadataRevisionRepair__Enabled=true to the Worker’s environment for the window (optional __BatchSize, default 500 rows; __RetryInterval, default 15 minutes; __Sources__0=<source> to limit the pass, default every repairable source) and recreate the Worker on its recorded compose chain. The bundle carries no compose variable for it: it is a one-time command, like the window’s image line. From pass v2 on (PCD-826 fact provenance) the pass runs a source only after the fact-provenance revision repair has recorded it, so enable both in the same start (next section).
  4. Watch docker logs -f stellaops-vulnerabilities-worker | grep 'revision repair': enabled: sources [...], a progress line every 20,000 examined rows, then per source complete: N candidate row(s), R repaired, Y without a later yielding revision, P held by fact provenance, U behind a refused revision, M moved by another writer. paused before its candidate read or paused after ... names the gate’s reason; the worker retries after RetryInterval and resumes from the rows that still hold an older revision. waits: the source normalization fence is held means an ingest cycle or a convergence pass holds the source; it retries the same way.
  5. Verify: one row per completed source in eventing.consumer_checkpoints with consumer_id = 'vuln.advisory-metadata-revision-repair#<pass>#<source>' (v2 on an image with the fact-provenance repair; v1 rows from an earlier window stay), and the measurement query above returns only the rows the log reported as not repaired.
  6. Remove the repair key (or leave it: a completed source is skipped on later starts).

Abort and rollback: remove the key and recreate the Worker (step 2 applies to that start), or stop it during the pass. Each batch commits on its own, so committed rows stay on their newer revision and the next enabled start resumes. The pass is forward-only: it does not record the revision a row held before, so only a restore point brings older metadata back.

What stays: rows held by fact provenance (the newest yielding revision backs no live fact of the row’s vulnerability, because the fact’s provenance was also left on the older revision) are not rewritten until the fact provenance is repaired (next section); pass v2 rewrites the ones that repair unblocks. Rows no later revision yields and the stellaops-mirror rows stay by design.

Fact-provenance revision repair

Symptom: after the advisory-metadata revision repair, its log still counts rows held by fact provenance: the newest revision that yields the row backs no live fact of the row’s vulnerability, because the facts’ provenance names an older retained revision of the same record.

Cause: before the VULN-B7 ruling (2026-08-26) a provenance origin kept the digest of the first document that asserted it, and before PCD-826 a fact-key convergence replay of a superseded revision moved an origin back to that revision. The routine pump moves an origin to the latest asserting document, and a replay no longer moves it back (module dossier, “Fact-provenance revision repair”), so the rows written before stay until this pass moves them.

Measure first (read-only). Per source, the provenance rows of live facts held by a superseded revision (a later retained revision of the same (source_key, upstream_ref) exists at or before the normalization checkpoint). This is the pass’s candidate set, not its write count: a row moves only to the newest later revision that asserts the same fact under the same origin with the stored content, which needs the current normalizer (a harness over a sample, as in the PCD-826 receipt). For a large source, split the read by substr(md5(r.upstream_ref), 1, 1) buckets:

BEGIN READ ONLY;
SELECT current_setting('transaction_read_only');
SET LOCAL lock_timeout = '3s'; SET LOCAL statement_timeout = '300s';
SET LOCAL max_parallel_workers_per_gather = 0; SET LOCAL jit = off;
WITH rd AS MATERIALIZED (
    SELECT r.digest, r.upstream_ref, COALESCE(r.last_observed_at, r.fetched_at) AS pos
    FROM ingest.raw_document r
    JOIN ingest.normalize_checkpoint c ON c.source_key = r.source_key
    WHERE r.source_key = 'osv' AND r.upstream_ref IS NOT NULL
      AND (COALESCE(r.last_observed_at, r.fetched_at), r.digest) <= (c.last_fetched_at, c.last_digest)),
newest AS MATERIALIZED (
    SELECT DISTINCT ON (upstream_ref) upstream_ref, digest, pos FROM rd ORDER BY upstream_ref, pos DESC, digest DESC),
held AS MATERIALIZED (
    SELECT rd.upstream_ref, rd.digest FROM rd JOIN newest n USING (upstream_ref)
    WHERE (n.pos, n.digest) > (rd.pos, rd.digest))
SELECT count(*) AS candidate_rows, count(DISTINCT h.digest) AS superseded_revisions
FROM held h
JOIN facts.fact_provenance p ON p.document_digest = h.digest
JOIN facts.fact_current f ON f.fact_id = p.fact_id AND f.tombstoned_at IS NULL;
COMMIT;

Procedure (vulnerabilities-worker only, in an authorized hub window; the Worker image must contain the PCD-826 fact-provenance repair):

  1. Steps 1 and 2 of the advisory-metadata revision repair apply unchanged (a fresh capacity receipt with writer_pause false; the key-convergence precondition for every Worker start).
  2. Add both keys to the Worker’s environment for the window and recreate the Worker on its recorded compose chain: Vulnerabilities__CorpusPublication__ProvenanceRevisionRepair__Enabled=true (optional __BatchSize, default 5,000 provenance rows; __RetryInterval, default 15 minutes; __Sources__0=<source>) and Vulnerabilities__CorpusPublication__MetadataRevisionRepair__Enabled=true. The provenance pass runs the sources in order; the metadata pass v2 of a source waits for the provenance pass of that source and retries every RetryInterval.
  3. Watch the Worker log for provenance revision repair and metadata revision repair lines: Fact-provenance revision repair enabled: sources [...], a progress line every 200,000 examined rows, then per source complete: N superseded revision(s), E row(s) examined, M moved, L not asserted by a later revision, C asserted with other content, U behind a refused revision, X changed by another writer. The metadata pass logs waits for the fact-provenance revision repair until then, followed by its own complete: line. paused lines and waits: the source normalization fence is held retry after RetryInterval.
  4. Verify read-only: one row per completed source in eventing.consumer_checkpoints for each of vuln.fact-provenance-revision-repair#v1#<source> and vuln.advisory-metadata-revision-repair#v2#<source>; the measurement query above returns about the rows the provenance log reported as not moved; the metadata measurement query returns the metadata log’s not-repaired rows; the next corpus publication follows the checkpoints.
  5. Remove the keys (or leave them: completed sources are skipped).

Space: no move is a HOT update (document_digest is indexed), so each writes a new heap tuple and a new entry in both indexes of facts.fact_provenance. Budget the rows the pass moves times (mean row width + 4 B) for the heap, plus up to about 200 B per row for the primary key and up to about 40 B per row for the digest index, against the headroom below the capacity stop line. The old versions stay until a vacuum: autovacuum starts at 50 + 0.2 x live rows (about 12 million on a 60-million-row table), more than one pass leaves, so pacing the pass does not let autovacuum reclaim space during it. Treat the growth as lasting; a plain VACUUM facts.fact_provenance after the pass makes the dead space reusable (the file does not shrink). The capacity writer gate before each batch stops the pass at the stop line.

Abort and rollback: remove the keys and recreate the Worker (the key-convergence precondition applies to that start), or stop it during the pass. Each batch commits on its own; the next enabled start resumes and reads the rows still on a superseded revision again. The pass is forward-only: it does not record the revision a row named before, so only a restore point brings it back. It writes no fact content, so fact sections of the next publication change only in their provenance digests.

What stays: provenance rows no later revision asserts (a fact a newer revision dropped or re-keyed; on SUSE CSAF most candidates are keys that the unrecorded suse-csaf-rpm-keys rule retires), rows whose newest asserting revision asserts other content (the stored content is then from an older normalization; the pass never changes content) and rows behind a refused revision.