Vulnerability data-plane separation investigation

Status: phase-1 findings and target proposal, extended with the phase-2 system-wide inventory; no database or runtime change has been applied.
Observed: 2026-07-22, running local Stella Ops compose estate.
Authority used: read-only PostgreSQL, Docker/configuration, source, migration, test, and log inspection.

Companion deliverables (2026-07-22 second pass):

Executive finding

The 91 GB database is called stellaops_platform, but Platform is not the writer of the dominant data. The platform schema is only 2,072,576 bytes (2024 kB). The physical database is large because Concelier, Excititor, VexHub, Policy, and other services are configured with the same STELLAOPS_POSTGRES_CONNECTION; their independently owned vuln, vex, vexhub, and concelier schemas therefore consume the Platform database and volume.

This is an ownership-boundary defect, not primarily a PostgreSQL-bloat problem. The target must move vulnerability source and serving data out of stellaops_platform into owner-specific physical databases. PostgreSQL maintenance is necessary only after that logical correction.

There is also an immediate correctness risk. The current Excititor retention worker deletes rows using identities that omit source-statement and applicability/range identity. It then deletes only orphan linksets; it does not transactionally rebuild surviving linksets or gate decisions. Live state proves that this has already produced stale derived data.

Validated live snapshot

SurfaceLive result
Docker volume compose_postgres-data99.25 GB
stellaops_platform database97,857,205,951 bytes (91 GB)
vuln schema76,210,806,784 bytes (71 GB)
vex schema9,625 MB
vexhub schema5,908 MB
concelier schema2,327 MB
platform schema2,072,576 bytes (2024 kB)
Physical host C: free45.3 GiB
PostgreSQL virtual filesystem free711 GiB; thin/sparse capacity, not physical host headroom

The four dominant serving relations are:

RelationRowsHeapIndexesTOAST/auxiliaryTotalTotal bytes/live row
vuln.issue_observations8,593,21316 GB5,074 MB1,156 MB22 GB2,805 B
vuln.issue_gate_decisions5,027,3918,762 MB5,738 MB4,054 MB18 GB3,870 B
vuln.issue_evidence_refs8,593,2136,308 MB6,038 MB1.8 MB12 GB1,507 B
vuln.issue_linksets5,027,3912,880 MB3,725 MB10 MB6,615 MB1,380 B

These four relations total about 58 GB and hold current live rows, indexes, and TOAST. A vacuum cannot remove that logical footprint.

The exact coherent integrity snapshot was:

observations                     8,593,213
evidence_refs                    8,593,213
linksets                         5,027,391
gate_decisions                   5,027,391
sum(linkset.observation_count)   9,791,122
linksets with wrong count          699,101
net stored observation excess    1,197,909
maximum excess in one linkset          308

There were no orphan linksets. The defect is stale surviving state, not merely undeleted orphans.

Why this is the third recurrence

The earlier repairs reduced symptoms without changing the physical ownership boundary:

  1. The May incident reduced a 306 GB database to 119 GB by deduplicating large payloads and dropping redundant columns/indexes.
  2. The June incident reached 420 GB. The repair added per-claim JSON deduplication, raw single-store changes, provider-run limits, and supersede retention. The unresolved vex.claims/vexhub.statements double store was later accepted as KEEP/optional.
  3. The July source-only mirror correctly stopped distributing the exploded serving projection, but explicitly retained the large local rebuild-on-import projection.
  4. The physical shared-database decision remained. The Concelier dossier says the services connect to stellaops_platform and treats schema isolation as sufficient. It is not sufficient for capacity, lifecycle, least-privilege, retention, or failure isolation.
  5. The June retention fix crossed module ownership: an Excititor worker now deletes Concelier vuln.issue_* rows. Its key is too coarse and its deletion is not reconciled with the derived graph.

The repeat pattern is therefore: optimize or purge the current representation, leave vulnerability data in the Platform physical database, rebuild the same exploded serving model, and rely on a guard attached to only one writer path. That pattern must end at the database and data-model boundaries.

Writer, lifecycle, retention, and reader trace

DataWriter/source lifecycleCurrent retention/reconciliationRead pathFinding
vex.claimsExcititor normalizes raw provider documentsRetention keeps one per (tenant, vulnerability, product, provider)Excititor APIs; Concelier projectorIdentity omits source statement and applicability/range; disjoint facts can be deleted
vexhub.statementsPostgresVexHubProjectionSink derives each statement from an Excititor claimRetention keeps one per (source, vulnerability, product)VexHub APIs; Concelier projectorDerived lineage is projected again as an independent observation; identity omits source_statement_id
vuln.issue_observationsConcelier projects advisory edges, VEX claims, and VexHub statementsExcititor retention keeps one per (tenant, issue, source_kind, source_key)Concelier issue API/linkset builderCross-owner destructive writer; source observation/applicability identity is omitted
vuln.issue_evidence_refsConcelier writes one reference per observationCascades from deleted observationsEvidence/read APIsFull reference exists for both claim and its derived VexHub representation
vuln.issue_linksetsConcelier aggregates observationsProjection rebuilds touched keys; retention deletes orphans onlyIssue API; gate projectorSurviving rows are stale after partial observation delete
vuln.issue_gate_decisionsConcelier gate projector consumes linksetsDirty queue reacts to linkset content-hash changesPolicy direct SQL; CLI compact exportRetention does not change/rebuild a surviving linkset, so gates do not see the logical change

There are no foreign keys crossing the vuln, vex, vexhub, or concelier schemas. Physical separation is blocked by application/configuration coupling, not cross-schema relational constraints. Direct consumers to move are Concelier projection/read services, Policy’s PostgresFindingsLookup, the CLI compact-vuln source, and mirror rebuild/export code.

Quantified causes

1. One source fact is represented repeatedly

vex.claims                           3,535,816
vexhub.statements                    3,535,816
vuln evidence kind vex-claim         3,535,816
vuln evidence kind vexhub-statement  3,535,816
vuln VEX observations                7,071,632

PostgresVexHubProjectionSink.BuildStatementRow derives VexHub fields from the claim: provider, document digest, vulnerability, product, status, justification, detail, and timestamps. Concelier then reads both and creates two full observations and two evidence references.

The coarse join (provider/source, document, vulnerability, product) returns 3,535,816 apparent pairs, but exact source-statement lineage exposes current damage:

claims missing their exact derived statement       424
statements without their exact originating claim   424
exact-lineage pairs with a different status          68
exact-lineage justification/detail mismatches          0

The 424/424 break is consistent with retention choosing a different statement/claim inside the too-coarse group. Equal counts do not prove provenance integrity.

One observation plus one evidence reference costs about 4.3 kB using total relation bytes/live rows. The second VEX representation therefore accounts for roughly 15 GB of direct observation/evidence storage at this corpus, plus repeated index, TOAST, aggregation, and replay cost. This is an estimate, not a cleanup quota; v2 must measure its result.

2. Full sweeps rewrite unchanged rows

All three observation upserts and all evidence upserts use unconditional ON CONFLICT ... DO UPDATE. Linkset rebuild also unconditionally updates every column and sets rebuilt_at = NOW(). vuln.issue_observations has a BEFORE UPDATE timestamp trigger. A logically idempotent sweep is physically non-idempotent.

The existing full-resweep test checks only unchanged row counts. It does not assert zero row versions or WAL. In a five-minute live measurement, update counters rose by 9,952 for observations/evidence and 8,579 for linksets. From the 09:35:16 UTC statistics reset to 11:39 UTC, the tables accumulated 82,659 observation/evidence updates and 71,288 linkset updates. Database WAL was about 19.1 GB in that interval; it includes all workloads and is not attributed solely to this projector.

Required guard:

ON CONFLICT (...) DO UPDATE SET ...
WHERE target.content_hash IS DISTINCT FROM excluded.content_hash
   OR target.is_deleted IS DISTINCT FROM excluded.is_deleted;

Linkset and gate writers need the same semantic-change rule. Rebuild timestamps belong in an audit/run record, not the hot row’s equality contract.

3. Retention deletes valid current facts and leaves derived state stale

Current retention partitions claims by (tenant, vulnerability_id, product_key, provider_id), statements by (source_id, vulnerability_id, product_key), and observations by (tenant_id, issue_key, source_kind, source_key). None includes applicability/range or stable source-statement identity.

A read-only characterization using that grouping proves the defect:

applicability range [3.0,4.0) -> row_number 1 -> kept
applicability range [1.0,2.0) -> row_number 2 -> marked for delete

Both ranges can be simultaneously valid. Age/latest is not a current-applicability rule.

The worker log shows two destructive sweeps after process starts on 2026-07-22:

05:48 UTC: deleted 31,698 vuln.issue_observations
09:42 UTC: deleted 15,616 vuln.issue_observations

The second followed a worker restart and five-minute initial delay. There is no durable last-run lease, so restarts can re-run the sweep. Because the first run’s effectively unbounded batch loop completed, the later 15,616 candidates are consistent with concurrent projection recreating rows; live insert/update counters support that inference, but the current schema has no audit record that can attribute every recreated row.

Deletes and orphan cleanup run as separate autocommit operations. The worker does not collect affected non-orphan issues, rebuild linksets/gates, and advance a durable watermark in one transaction. The 699,101 stale linksets are the live result.

4. Coordination is only partly durable

The incremental cursor is persisted, but the full-sweep watermark is an in-process dictionary. Although issue_projection_state.last_full_sweep_at exists, the cursor store neither reads nor writes it. On restart the service records now in memory and postpones a full sweep. There is no database lease/fencing token, so replicas can sweep concurrently.

Advisory ordering uses GREATEST(created_at, updated_at) and sees in-place changes. VEX claims use only created_at; VexHub statements use only ingested_at. An in-place VEX update is invisible until a full sweep, while restarts can postpone it.

5. Capacity controls miss the projection

The deployed Excititor guard is configured for an 80 GB database ceiling, 5,000 documents/run, and 250,000 claims/run. The database is already 91 GB. The guard exists per provider run and checks only during provider ingestion; it does not govern Concelier sweeps, VexHub repair, mirror rebuild, retention churn, or other paths. The Web process only displays the ceiling.

Every ingest/projection path needs a shared capacity contract.

6. Bloat and maintenance are secondary

Autovacuum is globally enabled. The large tables had no last_autovacuum/last_autoanalyze after the 09:35 UTC statistics reset. Default scale factors (vacuum 0.20, analyze 0.10) are too coarse for multi-million-row update-heavy tables. pgstattuple is not installed, and installing it would mutate the database, so exact bloat was not measured in phase 1.

The sizes are largely explained by live wide rows, JSON, TOAST, and indexes. A sample gate row averaged 2,207 inline bytes and 3,136 bytes across five JSON decisions before TOAST effects; the table also has 5.7 GB indexes and 4.0 GB TOAST. Tuning cannot remove an unnecessary live semantic copy.

Focused read-only reproductions

The phase-1 tests ran inside BEGIN ... READ ONLY transactions. They are intentionally diagnostic and must not be repurposed as cleanup SQL.

Disjoint-range characterization using the current retention partition:

WITH sample(vulnerability_id, product_key, provider_id, applicability_range, last_seen_at) AS (
  VALUES
    ('CVE-2099-0001', 'pkg:generic/acme/widget@1', 'provider-a', '[1.0,2.0)', TIMESTAMPTZ '2026-07-21 10:00:00+00'),
    ('CVE-2099-0001', 'pkg:generic/acme/widget@1', 'provider-a', '[3.0,4.0)', TIMESTAMPTZ '2026-07-22 10:00:00+00')
), ranked AS (
  SELECT *, row_number() OVER (
    PARTITION BY vulnerability_id, product_key, provider_id
    ORDER BY last_seen_at DESC) AS rn
  FROM sample
)
SELECT applicability_range, rn, rn > 1 AS current_retention_marks_for_delete
FROM ranked ORDER BY rn;

Coherent linkset-integrity test:

BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY;
WITH actual AS MATERIALIZED (
  SELECT tenant_id, issue_key, count(*)::bigint observation_count
  FROM vuln.issue_observations GROUP BY tenant_id, issue_key
), comparison AS MATERIALIZED (
  SELECT l.observation_count stored, a.observation_count actual
  FROM vuln.issue_linksets l JOIN actual a USING (tenant_id, issue_key)
)
SELECT count(*) FILTER (WHERE stored <> actual) mismatched_linksets,
       sum(stored - actual) net_stale_observation_excess,
       max(stored - actual) max_stale_excess
FROM comparison;
COMMIT;

Exact claim-to-derived-statement lineage test. The fallback identity mirrors ResolveSourceStatementId:

BEGIN TRANSACTION READ ONLY;
WITH expected AS (
  SELECT c.claim_hash, c.provider_id, c.document_digest,
         c.vulnerability_id, c.product_key, c.status, c.justification, c.detail,
         COALESCE(
           NULLIF(md.metadata_json->>'openvex.statement.id', ''),
           NULLIF(md.metadata_json->>'csaf.tracking.id', ''),
           'sha256:' || encode(sha256(convert_to(
             c.provider_id || '|' || c.document_digest || '|' ||
             c.vulnerability_id || '|' || c.product_key, 'UTF8')), 'hex')) expected_statement_id
  FROM vex.claims c
  LEFT JOIN vex.claim_metadata_dedup md ON md.content_hash = c.metadata_hash
), matched AS (
  SELECT e.*, s.id, s.status statement_status,
         s.justification statement_justification, s.status_notes
  FROM expected e
  LEFT JOIN vexhub.statements s
    ON s.source_id = e.provider_id
   AND s.source_statement_id = e.expected_statement_id
   AND s.vulnerability_id = e.vulnerability_id
   AND s.product_key = e.product_key
)
SELECT count(*) FILTER (WHERE id IS NULL) claims_missing_exact_statement,
       count(*) FILTER (WHERE id IS NOT NULL AND status IS DISTINCT FROM statement_status) exact_status_mismatches,
       count(*) FILTER (WHERE id IS NOT NULL AND justification IS DISTINCT FROM statement_justification) exact_justification_mismatches,
       count(*) FILTER (WHERE id IS NOT NULL AND detail IS DISTINCT FROM status_notes) exact_detail_mismatches
FROM matched;
COMMIT;

The physical no-op test used two snapshots of pg_stat_user_tables.n_tup_upd around a five-minute interval while source counts were stable. The v2 acceptance test must use an isolated database and a stricter per-test audit/WAL boundary rather than shared live statistics.

Target physical ownership

DatabaseOwnerContents
stellaops_platformPlatformPlatform control-plane only; no vulnerability corpus/projection
stellaops_concelierConcelierImmutable advisory sources, normalized source facts, outbox/cursors
stellaops_excititorExcititorImmutable VEX documents, normalized fact revisions, verification, outbox/cursors
stellaops_vexhubVexHubPublic/global distribution and verification read model if retained as a durable boundary
stellaops_vulnVulnerability serving plane, owned by ConcelierCurrent semantic facts, provenance, overlays, hot linksets/gates, leased checkpoints, separate history

stellaops_vuln is deliberately not Platform. If architecture colocates it operationally with Excititor, it must still be a dedicated physical database/role; schema-only separation inside stellaops_platform is rejected.

flowchart LR
    subgraph CDB["stellaops_concelier"]
        AR["Immutable advisory documents"] --> AF["Normalized advisory facts"] --> AO["Transactional outbox"]
    end
    subgraph EDB["stellaops_excititor"]
        VR["Immutable VEX documents"] --> VF["Normalized VEX fact revisions"] --> VO["Transactional outbox"]
    end
    subgraph HDB["stellaops_vexhub"]
        VH["Distribution and verification view"]
    end
    subgraph VDB["stellaops_vuln"]
        PF["Current semantic facts"]
        PR["Provenance and lineage"] --> PF
        TO["Tenant overlays"] --> LS["Hot issue linksets"] --> GD["Hot gate decisions"]
        PF --> LS
        CP["Leased checkpoints and fencing"] --> PF
        PF --> HI["Partitioned history/archive"]
    end
    AO -->|"idempotent changes/tombstones"| CP
    VO -->|"idempotent changes/tombstones"| CP
    VO --> VH
    VH -. "lineage/verification, not a second fact" .-> PR
    GD --> POL["Policy/API/CLI readers"]
    PLAT["stellaops_platform"] ~~~ VDB

Canonical identity and deduplication

Use two identities:

  1. Issue/linkset identity: (tenant scope, canonical vulnerability id, canonical product identity). This coarse key aggregates facts.
  2. Semantic fact identity: (issuer/source, stable source_statement_id, canonical vulnerability id, canonical product identity, normalized applicability selector/range hash). Revision/content hash is a value of the stable fact, not its sole identity.

Rules:

Transactional reconciliation

  1. The source commits a normalized revision/tombstone and monotonically ordered outbox record in one source transaction.
  2. One destination projector holds a renewable lease with fencing token per stream.
  3. In one stellaops_vuln transaction it applies content-guarded fact/provenance changes, tombstones removed facts, collects affected issues, rebuilds linksets/gates, and advances the checkpoint.
  4. Failure rolls back the destination transaction/checkpoint. Replay is safe through deterministic identities/hashes.
  5. Full reconciliation uses a source generation plus completed marker. Only after completion may absent facts be tombstoned. Age alone never means “not applicable.”
  6. Removing one provenance record does not tombstone a fact while another active provenance remains. Removing the last does and rebuilds derived rows transactionally.

Phased remediation and rollback

Phase 0 — urgent safeguards; runtime approval required

Phase 1 — stop new corruption/write amplification in v1

Rollback is forward-only: disable new projectors/outboxes and return readers to v1. Do not remove columns/tables or edit applied migrations.

Phase 2 — side-by-side v2

Rollback stops v2 consumers/projectors and leaves v2 intact; v1 readers continue.

Phase 3 — validate and cut over

Required for two complete source generations plus agreed soak:

Cutover drains to a recorded watermark, switches readers reversibly, and keeps v1 read-only for rollback. Rollback switches readers, not migrations.

Phase 4 — explicit retirement/reclaim

Only after rollback window, verified backup, signed reconciliation, and separate destructive approval:

Capacity gate

NeedTemporary capacity
v2 physical build before measured dedup60–100 GB
WAL, indexes, sort/temp, checkpoint/replication headroom40–80 GB
verified full backup/restore artifactabout 100 GB, preferably off-host
contingency40–60 GB

If backup is local, reserve roughly 240–340 GB additional physical capacity. With off-host backup, reserve at least 140–240 GB on the database host plus off-host backup capacity. Current C: free space is 45.3 GiB, so the gate is red despite the sparse Linux filesystem showing 711 GiB.

PostgreSQL after the model correction

Initial values to validate on v2, not apply in phase 1:

Metrics and limits for every path

Cover advisory/VEX ingest, VexHub project/repair, issue projection, mirror rebuild, reconciliation, and archive:

Required tests

ProofRequired test
No valid range lostTwo same-issuer/vulnerability/product disjoint ranges survive ingest, replay, reconciliation, and history compaction; deleting one leaves the other
Provenance retainedExcititor claim + derived VexHub statement yields one fact/two lineage records; external VexHub-only remains independent
Current break detectedFixtures reproduce 424/424 lineage and 68 status-drift classes and fail until repaired/tombstoned
Deletes reconcileRemove one provenance, last provenance, and one of several ranges; assert exact fact/linkset/gate in one transaction; injected failure rolls back checkpoint too
Replay idempotentSecond generation changes zero semantic/hot rows and only bounded checkpoint/audit state
No-op avoids writesPostgreSQL stats/WAL or trigger audit proves zero hot-table updates
Single writerTwo instances contend; only leased fencing token commits; expired owner cannot commit
Update visibilityIn-place revision emits outbox and reaches v2 without full sweep; restart resumes durable checkpoint
Side-by-side parityFull corpus counts, identities, ranges, provenance, linksets, gates, Policy/API/CLI, intentional-difference ledger
Backup/rollbackRestore fresh v2, cut over, inject failure, switch to untouched v1, verify both directions

Existing tests are insufficient: full-resweep asserts counts, not writes; retention has no focused proof for disjoint ranges/source statements or downstream rebuild.

System-wide inventory (phase 2, 2026-07-22)

The second read-only pass widened the evidence from the vulnerability plane to the whole estate:

Decisions requiring approval (VDPS-D1 … VDPS-D12)

Direction revised 2026-07-22 (owner review): the target moved from four separate databases to a single consolidated StellaOps.Vulnerabilities microservice with one database, a tenant-free hub, per-tenant gates relocated to a Policy-owned projection, and a fresh rebuild from sources with no data preservation (replacing the side-by-side backfill/parity plan this document’s phase-1 sections describe). The findings above remain valid evidence; the current decision set D1–D12, with proposed outcomes, alternatives, and consequences, lives in ADR-039 rev-2, with the target model in vulnerability-data-plane-v2-design.md.

Summary of the current set: D1 out-of-platform; D2 consolidation into StellaOps.Vulnerabilities (one service, one DB, plugins for sources and mirrors, lookup APIs); D3 per-tenant gates move to Policy, CLI export reads the hub’s global API; D4 fact/issue identity with applicability; D5 outbox/lease standard; D6 retention freeze only (no v1 repair); D7 fresh re-ingest with coverage-based gates; D8 ~100 GB dedicated vulndata volume; D9 separate destructive decommission (with recommended pre-drop snapshot); D10 tenancy/IdP owned by Authority (kills the platform-DB coupling); D11 doctor-as-plugins-per-service; D12 StellaOps.<Service>.Persistence naming standard.

Evidence pointers