Vulnerability Operations Runbook

Audience: Platform on-call and operators triaging vulnerability scanning, advisory-feed, and findings problems in a Stella Ops deployment.

Purpose: Provide a fast, deterministic path to triage, mitigate, and evidence vulnerability-pipeline issues — invalid scanner packs or advisory feeds, unhealthy Scanner / Findings / vulnerabilities-web / vulnerabilities-worker services, and stuck or failed jobs — while preserving reproducibility (hashes and digests) end to end.

Supported Scope

This runbook covers vulnerability operations for Docker Compose, Offline Kit, signed release manifests, and host/service-manager deployments.

Kubernetes and Helm mock overlays are retired. Do not use helm template, helm upgrade, helm rollback, or kubectl scaling/log commands for Stella Ops vulnerability operations.

Current owner: src/Vulnerabilities; the old Concelier/VexLens services and schemas are retired. Related runbooks: VEX Operations · Scanner worker not processing jobs

Triage

1. Capture the affected tenant, environment, scanner pack, advisory feed bundle, and release manifest digest.
2. Verify scanner pack and advisory feed hashes against the signed release or Offline Kit manifest.
3. Check Scanner, Findings, vulnerabilities-web, and vulnerabilities-worker health through supported service endpoints and host logs.
4. Review queue depth and failed job evidence from the Stella Ops UI/API.

Mitigation

1. Pause affected vulnerability jobs if the feed or scanner pack is invalid.
2. Re-import the last known-good Offline Kit feed bundle when feed corruption is confirmed.
3. Restart only the affected live service (vulnerabilities-web, vulnerabilities-worker, Scanner, or Findings) through Docker Compose or the host service manager.
4. Re-run the failing vulnerability job and compare output digests.

Evidence

• Feed bundle id and hash.
• Scanner pack id and hash.
• Job id and output digest.
• Release manifest digest.
• Health-check and host-log excerpts.

Retained corpus publication bring-up

vulnerabilities-worker publishes at most one retained corpus publication per ingest cycle (IngestScheduler -> CorpusArtifactProducer.PublishCompletedGenerationAsync) into the compose_vulnerabilities-corpus-publications volume; vulnerabilities-web serves it read-only at /api/vulnerabilities/v1/corpus/publications/**. The cadence was one publication per completed source generation until the owner ruling of 2026-08-27 (sprint 003 VULN-B8 (a)); it is now taken once, after every enabled source and the EPSS snapshot have been attempted, and is stamped with the last source in configured EnabledSources order that completed a publishable (non-binary-corpus) generation in that cycle. A cycle in which nothing completed publishes nothing.

The scheduler logs the decision explicitly — grep the worker for either line:

Corpus publication for this cycle: ONE artifact corpus-publication:<hex> stamped '<source>' generation <N> (<k> of <n> enabled sources completed a publishable generation).
No corpus publication for this cycle: none of the <n> enabled sources completed a publishable generation.

A failed publication is a CYCLE outcome, not a source outcome: it appears as Ingest cycle corpus-publication: FAILED (...) and never marks a healthy source failed. Alongside those, watch runtime.corpus_publication / runtime.corpus_publication_current, the volume’s .tmp/ growth, and the worker’s Ingest run for <source> completed generation N / Ingest cycle failed for source lines, which bracket each source attempt.

Two blockers measured on the first live bring-up (2026-08-26) and their fixes:

1. Volume root owned by root. Both hub images run as uid 10001 and carry no /app/data, so a volume first created by any container mount is root:root 755 and every attempt fails with UnauthorizedAccessException: Access to the path '/app/data/corpus-publications/.control' is denied. Fix once, while the volume is empty: docker run --rm --user 0:0 --entrypoint sh -v compose_vulnerabilities-corpus-publications:/v stellaops/vulnerabilities-worker:dev -c 'chown 10001:10001 /v', then docker restart stellaops-vulnerabilities-worker (the boot cycle re-fires the trigger). Verify from inside the worker: ls -ld /app/data/corpus-publications shows stella stella.
2. A stored fact identity string with boundary whitespace (upstream OSV alias "CVE-2020-13665 ") failed the advisory-metadata wire contract (CorpusWireFormatException: Advisory-metadata field 'vulnerabilityId' must be non-empty and have no boundary whitespace) after ~10 GB of the other sections had streamed. VulnFact now trims identity strings on construction and migration 003_fact_identity_representation_convergence.sql converges rows written before that guard at the next hub boot. Detect a recurrence with SELECT count(*) FROM facts.fact_current WHERE vulnerability_id ~ '^\s|\s$' OR product_key ~ '^\s|\s$' (expected 0); never hand-edit hub tables.

Success is GET /api/vulnerabilities/v1/corpus/publications/current = 200 with a manifest digest, one runtime.corpus_publication_current row, and one eventing.outbox row on stream vuln.corpus per publication. Before the first run check free space on the volume host and the database host: the pre-compact publication is ~10.7 GiB; compact inputs and source provenance bring the 2026-09-09 measured estimate to ~12.3 GiB (cap 16 GiB, current + previous retained under 32 GiB).

Measured first live publication (2026-08-26, kev generation 45): ~16 min from generation completion to readability; 10,418,775,281 B across five sections (matcher-rows 4.20 GB, consensus-inputs 5.96 GB, exploit-evidence 40.6 MB, advisory-metadata 221 MB, reachability-sinks 0); the store discards the parts of a failed attempt before the next one starts (nothing is reused across attempts), and each completed, pumped source generation publishes again, so a full cycle can stream several publications back to back. docker restart of the worker re-fires the trigger at boot via kev.

Distro binary-corpus bring-up (build-id plane)

Compressed index containers, fixed 2026-09-04 (D-BIN11-8) — point the repository at whichever index your mirror publishes. The stage now strips the transport container off the index before parsing it, and it decides on the BYTES rather than the URL, so an endpoint with no extension, a redirect, or a proxy-renamed file all read correctly. Accepted: plain text, gzip, xz, bzip2, and a tar (compressed or not) whose APKINDEX / Packages member holds the index — which is what APKINDEX.tar.gz is. A tarball holding exactly one file is accepted whatever that file is called. Refused by name, with the reason in the log rather than a silent empty result: .zst, because this build ships no zstd decoder.

Until 2026-09-04 none of that existed and apk-index could not read a single real Alpine mirror (D-BIN11-7): every cycle ended in Repository index kind 'apk-index' enumerated zero packages. If you are running an image built before that date, this is still true, and deb-packages behaves the same way against a Packages.gz/.xz URL — on an older image, use a PLAIN Packages index. Either way the stage fails closed: zero rows written, the checkpoint held behind the document, so nothing needs cleaning up and an upgraded build re-reads the same stored document.

Still unsupported, and this one is a capability gap rather than a container: rpm-repomd. repomd.xml enumerates no packages at all — it only locates primary.xml.gz — so the stage refuses it loudly instead of recording an empty repository as analysed.

Alpine package URLs, fixed 2026-09-05 (D-BIN11-9) — an image built between the two fixes enumerates the index and then 404s every package. The symptom is unmistakable and it is not a mirror problem: the log reads Index …/main/x86_64/APKINDEX.tar.gz enumerates N packages and is then followed by one Skipping <pkg> <ver>: <uri> answered 404. per package, with the architecture DOUBLED in the URI (…/main/x86_64/x86_64/<name>-<version>.apk). binary_identity and binary_package_map stay at zero rows and the footprint does not move. An Alpine repository publishes APKINDEX.tar.gz and the .apk files in the SAME directory, so the package path is resolved beside the index; the stage was re-stating the architecture that the index URL already ends in. Nothing to configure and nothing to clean up — the run writes no rows and holds its checkpoint behind the document, so an upgraded build re-reads the same stored index. Debian is unaffected: a Packages stanza states a repository-root-relative Filename, which is rebased to the segment before /dists/ and always was.

Note for whoever repeats the run: those per-package fetches are rate-limited as of 2026-09-07 (VULN-B12) and were not before. The payload client now sends the hub User-Agent and paces itself per host at a shipped 2.0 rps initial / 0.25 floor / 8.0 ceiling, AIMD, honouring Retry-After and retrying a 429/503 instead of counting the package as skipped. An image built before that date issues one anonymous request per package as fast as it can.

The pacing is per HOST and is shared with the index fetch, so a bounded pass of 8 x 2,000 packages spends on the order of an hour on politeness alone. An estate serving the corpus from its own mirror should raise that host’s cap rather than wait: Vulnerabilities__Ingestion__Throttling__HostOverrides__<host>__InitialRps. Pointing the repository at your own mirror (below) is the standing recommendation for this source anyway.

A truncated index no longer passes as a short one, and an index nobody can vouch for no longer authorizes deletion (VULN-B11, 2026-09-07). A repository index cut off mid-transfer is a well-formed PREFIX: gzip decompresses it without an exception, the parser reads it without an exception, and it enumerates a real but incomplete subset. Nothing below the connector can tell it from a genuinely small repository — and a completed Full-inventory generation is what authorizes D4 absence-tombstones, so that prefix could authorize deleting binaries that still exist upstream.

Declare an anchor per repository and the connector checks the bytes against it:

Vulnerabilities__Ingestion__Sources__DistroBinaryCorpus__Repositories__0__IndexSha256: "<hex>"
Vulnerabilities__Ingestion__Sources__DistroBinaryCorpus__Repositories__0__IndexSizeBytes: "<bytes>"

Either one is enough. IndexSha256 is the stronger of the two: it is the only check that catches a mirror holding a truncated FILE, because that transfer is complete and correctly framed and no transport check can see it. IndexSizeBytes catches truncation but not substitution, and is offered because a directory listing gives a length for free. The transport’s own Content-Length is always checked and needs no configuration.

If you declare neither, nothing breaks — the estate loses absence authority, not ingestion. The run still fetches, still stores the index, and still fills the build-id plane, because build-id facts are additive and true whether or not the enumeration was complete. What changes is that the generation is recorded inventory_scope = 'delta' instead of 'full', and by the connector SPI’s own rule a Delta generation may never authorize an absence-tombstone. The worker says so once per index per run:

Distro binary-corpus index <uri> declares no IndexSha256 or IndexSizeBytes, so a truncated index
cannot be told from a short one. The run is recorded as a Delta inventory and can NOT authorize
absence-tombstones.

An air-gapped or pinned corpus should turn that degradation into a refusal, because there an unanchored index is a misconfiguration rather than a fact of life: Vulnerabilities__Ingestion__Sources__DistroBinaryCorpus__RequireDeclaredIntegrity: "true". It is OFF by default because a LIVE mirror re-publishes its index whenever the repository changes, so a pinned digest would have to be re-declared daily and defaulting to a refusal would break every unpinned estate on upgrade.

Check which you are getting:

SELECT generation, state, inventory_scope, completed_at
FROM runtime.source_generation
WHERE source_key = 'distro-binary-corpus'
ORDER BY generation DESC LIMIT 5;

distro-binary-corpus is the source that fills the hub’s build-id plane (binaries.binary_identity + binaries.binary_package_map). It is opt-in per estate, it is absent from every default source set, and it fails closed when unconfigured — a corpus-less deployment is fully functional and its exported build-id index is simply empty.

Point it at your own mirror, not an upstream CDN

Owner ruling 2026-09-04: this product ships no default host and makes no default outbound call, so no repository URL appears in any tracked file. The recommended target is the mirror the estate already runs, not dl-cdn.alpinelinux.org or any other public CDN. One bounded pass fetches thousands of package archives, which is mirror-shaped traffic; a mirror you pin also gives the same corpus twice, which a CDN moving under you does not. An air-gapped estate serves APKINDEX and the .apk files from its own host and needs no outbound access at all.

The configuration is deliberately split in two:

Apply them after docker-compose.vulnerabilities.g2.override.yml, local file last. Compose merges environment maps by key, so EnabledSources__8 extends G2’s 0..7 rather than replacing it.

What “fails closed” means here, precisely

• No repository configured — DistroBinaryCorpusSourceConnector throws and the generation is abandoned. It is never recorded as a completed empty Full run, because a completed Full generation is what authorizes D4 absence-tombstones.
• A repository with no IndexUri— throws naming the missing key.
• So applying the tracked overlay alone leaves the source FAILED every cycle with a message that names what is missing. That is the intended state, not a defect to fix by pasting a public mirror into the tracked file.
• A payload that cannot be fetched degrades per package, not per run. A blocked or 404 .apk is logged and counted as skipped; the loop continues. An egress policy that blocks the mirror therefore produces a completed run with zero binaries, not a failed hub.

Runtime cost, stated plainly

Enabling this source makes vulnerabilities-worker perform outbound HTTP from the customer’s own deployment, inline in the ingest cycle (IngestScheduler runs the analysis stage synchronously after the source’s generation completes). Package bytes are transient: each payload spools to a FileOptions.DeleteOnClose temp file and cannot outlive its handle, and package archives never become ingest.raw_document rows — that split is the design §9.1 capacity invariant. What is retained is the derived identity: build-id, file digest, format/architecture/type/stripped flag, in-package path, and the package’s purl/version/distro coordinates.

Peak transient disk is capped per package, independently of how many packages a pass analyses: MaxPackageBytes 512 MiB, MaxEntryBytes 64 MiB, MaxUncompressedBytes 1 GiB, MaxEntries 50000.

The capacity lever

Vulnerabilities:Binaries:Analysis:MaxPackagesPerDocument (env Vulnerabilities__Binaries__Analysis__MaxPackagesPerDocument) bounds the packages analysed per index document. Default 250.

It bounds a serial loop — not memory, not document size, not any database column or wire payload. Because per-package disk is capped separately, raising it does not raise peak transient disk; the cost it buys is wall-clock inside the ingest cycle. Never set it to 0 (unbounded) against a Debian Packages index, which enumerates roughly 60k packages.

Reading the result

The worker logs one line per source per cycle; the binary corpus reports corpus counters rather than facts=, because it writes no advisory facts:

Ingest cycle distro-binary-corpus: ok (seen=1, new=1, indexes=1, packages=N (skipped=M), binaries=B, corpusRows=R).

The stage logs its own summary, and logs the truncation explicitly when the lever bites:

Binary corpus analysis for distro-binary-corpus generation <G>: <D> indexes, <N> packages (<M> skipped), <B> binaries, <I> identity and <P> package-map row versions written.
Index <uri> enumerates <T> packages; analysing the first <L> (Vulnerabilities:Binaries:Analysis:MaxPackagesPerDocument).

Measure the plane against design §9.1’s budget with the doctor check on vulnerabilities-web(not the worker, which hosts no doctor checks):

GET /doctor/vulnerabilities-web/checks   ->  checkId "doctor.vulnerabilities.binaries.corpus-footprint"

It requires the ops.health scope (vulnerabilities:health:read policy) and returns footprint.bytes, budget.bytes (Vulnerabilities:Binaries:FootprintBudgetBytes, default 25 GiB = 26843545600) and scope.lever. The check refuses to answer at all unless the measured database actually holds the folded plane, so it can never report “within budget” about the retiring schemas of the same name on the same server.

Advisory-metadata backfill

Symptom: publications succeed but every advisory-metadata row carries null title/severity/ CVSS, and SELECT count(*) FROM facts.advisory_metadata_input is far below SELECT count(DISTINCT vulnerability_id) FROM facts.fact_current WHERE tombstoned_at IS NULL. Measured on 2026-08-26: 0 candidate rows against 669,406 live vulnerabilities.

Cause: migration 002 created the metadata projection after the estate had already normalized its documents, and NormalizationPump writes candidates only for documents past its per-source checkpoint. On a quiet estate (“N seen, 0 new” per cycle) nothing revisits the backlog, so the projection stays empty indefinitely. The standing mechanism is unchanged — the pump writes every NEW document’s candidate in the fact transaction; the backlog needs the one-time pass below.

Procedure (vulnerabilities-worker only; the pass is the hub’s own code path — the same CorpusAdvisoryMetadataInputNormalizer + CorpusAdvisoryMetadataInputWriter the pump uses, over the retained ingest.raw_document payloads, in the pump’s own (fetched_at, digest) order):

1. Set VULN_METADATA_BACKFILL_ENABLED=true in the compose .env (optional VULN_METADATA_BACKFILL_BATCH_SIZE, default 200 documents per transaction) and recreate the worker on its recorded compose chain (--force-recreate --no-deps vulnerabilities-worker). Migration 004_advisory_metadata_backfill_checkpoint converges at boot.

2. Watch docker logs -f stellaops-vulnerabilities-worker | grep 'Advisory-metadata backfill': enabled: sources [...], then a progress line every 5,000 documents (docs/min), then ... for <source> complete: N live documents scanned, M candidate rows written, and finally pass finished ...; idling until the next worker start. Progress and completion per source are also in facts.advisory_metadata_backfill (documents_scanned, candidates_written, completed_at).

3. Verify coverage: SELECT count(*), count(DISTINCT vulnerability_id) FROM facts.advisory_metadata_input against the live-vulnerability count. Vulnerabilities known only through alpine-secdb / debian-security-tracker have no metadata authority and stay all-null by contract — confirm that is ALL the uncovered set is, rather than assuming it, by grouping the uncovered vulnerabilities’ live facts by issuer_id: only those two issuers may appear. Then check that every candidate is reachable by metadata-selection/v1. Use the selector’s exact predicate — it also requires the live fact to be for the same vulnerability, so a query that joins only on fact_id is a weaker test and will pass rows the selector drops:

   SET max_parallel_workers_per_gather = 0;   -- required, see below
   SELECT count(*) FROM facts.advisory_metadata_input i
   WHERE NOT EXISTS (
       SELECT 1 FROM facts.fact_current live
       JOIN facts.fact_provenance p ON p.fact_id = live.fact_id
       WHERE live.vulnerability_id = i.vulnerability_id
         AND live.tombstoned_at IS NULL
         AND p.document_digest = i.document_digest);   -- expected 0
   

   SET max_parallel_workers_per_gather = 0 is not optional. The verification query hits the same wall the walk itself did: the planner turns the anti-join into a parallel hash join over the 33M-row facts.fact_provenance table and the database container’s /dev/shm runs out (53100: could not resize shared memory segment). With parallelism off it is an index-driven plan and completes.

4. The next completed source generation publishes with the populated section; fetch it by handle from /api/vulnerabilities/v1/corpus/publications/current and spot-check a known CVE. Range the fetch. A populated advisory-metadata section is ~934 MB, and an UNRANGED GET through stellaops-router-gateway returns 502 urn:stellaops:router:invalid-upstream-response (the gateway materialises the whole upstream response into one transport frame). Measured 2026-08-27: <= 48 MiB ranged -> 206, 64 MiB -> 504 after 30 s, >= 96 MiB -> 502. Use Range: bytes=<start>-<end> at 16 MiB and concatenate; Accept-Ranges, Content-Range, ETag/If-Range and X-StellaOps-Content-Digest are all correct, so the concatenated bytes hash to the catalogue’s section_digest.

5. Set the variable back to false (or leave it: a re-run scans only the window the pump has since advanced and writes nothing — the writer’s content guard suppresses identical rows).

Scope and cost: only documents whose digest still backs a live, non-tombstoned fact are read (superseded record versions are skipped without detoasting their payload); the pass never advances past a source’s normalization checkpoint. On the 2026-08-26 estate that is ~940k documents and ~25 GB of payload reads (osv 9.6 GB, redhat-csaf 6 GB, suse-csaf 6 GB, cve 1.7 GB, nvd 1.55 GB); expect hours, not minutes, and no additional disk beyond the projection rows. Interrupting the worker is safe: each batch commits with its checkpoint and a restart resumes. To force a full re-walk of one source, DELETE FROM facts.advisory_metadata_backfill WHERE source_key = '<key>' (a checkpoint row, not data — the only hand-run statement this procedure allows).

Measured outcome of the one live pass (2026-08-26 15:08Z -> 2026-08-27 06:23:28Z, verified 2026-08-27 08:30Z; re-verify with the step-3 queries). facts.advisory_metadata_input = 1,247,490 rows over 668,228 distinct vulnerabilities, 1374 MB against 669,579 live vulnerabilities = 99.798%; per source cve 203,383 / nvd 312,274 / osv 349,370 / redhat-csaf 112,455 / suse-csaf 270,008. Unreachable candidates 0. The 1,351 uncovered grouped by issuer_id are exactly debian-security-tracker (1,209) and alpine-secdb (173) — no metadata authority, all-null by contract — so coverage of the derivable population is 668,228 / 668,228 = 100%. The first publication after the pass carried a 979,698,214-byte advisory-metadata section against 221,164,791 B for the all-null one that preceded the backfill. Wall time was dominated by the payload reads, not the writes: cve 1 h 09 m, nvd 1 h 24 m, osv 1 h 32 m, with redhat-csaf/suse-csaf behind them. Two source-data and infrastructure traps hit during that pass are recorded in sprint 003’s Decisions & Risks (invalid UTF-8 in a SUSE CSAF note, handled by treating an unreadable string token as absent; and the /dev/shm parallel-hash wall above).