Vulnerability data-plane separation investigation
Status: phase-1 findings and target proposal, extended with the phase-2 system-wide inventory; no database or runtime change has been applied.
Observed: 2026-07-22, running local Stella Ops compose estate.
Authority used: read-only PostgreSQL, Docker/configuration, source, migration, test, and log inspection.
Companion deliverables (2026-07-22 second pass):
- Proposed system-wide policy: ADR-039 — database-per-service ownership
- Full deployable→database inventory and read-contract register: database-ownership-matrix.md
- Target model, ERD, storage budget, migration and test plan: vulnerability-data-plane-v2-design.md
Executive finding
The 91 GB database is called stellaops_platform, but Platform is not the writer of the dominant data. The platform schema is only 2,072,576 bytes (2024 kB). The physical database is large because Concelier, Excititor, VexHub, Policy, and other services are configured with the same STELLAOPS_POSTGRES_CONNECTION; their independently owned vuln, vex, vexhub, and concelier schemas therefore consume the Platform database and volume.
This is an ownership-boundary defect, not primarily a PostgreSQL-bloat problem. The target must move vulnerability source and serving data out of stellaops_platform into owner-specific physical databases. PostgreSQL maintenance is necessary only after that logical correction.
There is also an immediate correctness risk. The current Excititor retention worker deletes rows using identities that omit source-statement and applicability/range identity. It then deletes only orphan linksets; it does not transactionally rebuild surviving linksets or gate decisions. Live state proves that this has already produced stale derived data.
Validated live snapshot
| Surface | Live result |
|---|---|
Docker volume compose_postgres-data | 99.25 GB |
stellaops_platform database | 97,857,205,951 bytes (91 GB) |
vuln schema | 76,210,806,784 bytes (71 GB) |
vex schema | 9,625 MB |
vexhub schema | 5,908 MB |
concelier schema | 2,327 MB |
platform schema | 2,072,576 bytes (2024 kB) |
Physical host C: free | 45.3 GiB |
| PostgreSQL virtual filesystem free | 711 GiB; thin/sparse capacity, not physical host headroom |
The four dominant serving relations are:
| Relation | Rows | Heap | Indexes | TOAST/auxiliary | Total | Total bytes/live row |
|---|---|---|---|---|---|---|
vuln.issue_observations | 8,593,213 | 16 GB | 5,074 MB | 1,156 MB | 22 GB | 2,805 B |
vuln.issue_gate_decisions | 5,027,391 | 8,762 MB | 5,738 MB | 4,054 MB | 18 GB | 3,870 B |
vuln.issue_evidence_refs | 8,593,213 | 6,308 MB | 6,038 MB | 1.8 MB | 12 GB | 1,507 B |
vuln.issue_linksets | 5,027,391 | 2,880 MB | 3,725 MB | 10 MB | 6,615 MB | 1,380 B |
These four relations total about 58 GB and hold current live rows, indexes, and TOAST. A vacuum cannot remove that logical footprint.
The exact coherent integrity snapshot was:
observations 8,593,213
evidence_refs 8,593,213
linksets 5,027,391
gate_decisions 5,027,391
sum(linkset.observation_count) 9,791,122
linksets with wrong count 699,101
net stored observation excess 1,197,909
maximum excess in one linkset 308
There were no orphan linksets. The defect is stale surviving state, not merely undeleted orphans.
Why this is the third recurrence
The earlier repairs reduced symptoms without changing the physical ownership boundary:
- The May incident reduced a 306 GB database to 119 GB by deduplicating large payloads and dropping redundant columns/indexes.
- The June incident reached 420 GB. The repair added per-claim JSON deduplication, raw single-store changes, provider-run limits, and supersede retention. The unresolved
vex.claims/vexhub.statementsdouble store was later accepted asKEEP/optional. - The July source-only mirror correctly stopped distributing the exploded serving projection, but explicitly retained the large local rebuild-on-import projection.
- The physical shared-database decision remained. The Concelier dossier says the services connect to
stellaops_platformand treats schema isolation as sufficient. It is not sufficient for capacity, lifecycle, least-privilege, retention, or failure isolation. - The June retention fix crossed module ownership: an Excititor worker now deletes Concelier
vuln.issue_*rows. Its key is too coarse and its deletion is not reconciled with the derived graph.
The repeat pattern is therefore: optimize or purge the current representation, leave vulnerability data in the Platform physical database, rebuild the same exploded serving model, and rely on a guard attached to only one writer path. That pattern must end at the database and data-model boundaries.
Writer, lifecycle, retention, and reader trace
| Data | Writer/source lifecycle | Current retention/reconciliation | Read path | Finding |
|---|---|---|---|---|
vex.claims | Excititor normalizes raw provider documents | Retention keeps one per (tenant, vulnerability, product, provider) | Excititor APIs; Concelier projector | Identity omits source statement and applicability/range; disjoint facts can be deleted |
vexhub.statements | PostgresVexHubProjectionSink derives each statement from an Excititor claim | Retention keeps one per (source, vulnerability, product) | VexHub APIs; Concelier projector | Derived lineage is projected again as an independent observation; identity omits source_statement_id |
vuln.issue_observations | Concelier projects advisory edges, VEX claims, and VexHub statements | Excititor retention keeps one per (tenant, issue, source_kind, source_key) | Concelier issue API/linkset builder | Cross-owner destructive writer; source observation/applicability identity is omitted |
vuln.issue_evidence_refs | Concelier writes one reference per observation | Cascades from deleted observations | Evidence/read APIs | Full reference exists for both claim and its derived VexHub representation |
vuln.issue_linksets | Concelier aggregates observations | Projection rebuilds touched keys; retention deletes orphans only | Issue API; gate projector | Surviving rows are stale after partial observation delete |
vuln.issue_gate_decisions | Concelier gate projector consumes linksets | Dirty queue reacts to linkset content-hash changes | Policy direct SQL; CLI compact export | Retention does not change/rebuild a surviving linkset, so gates do not see the logical change |
There are no foreign keys crossing the vuln, vex, vexhub, or concelier schemas. Physical separation is blocked by application/configuration coupling, not cross-schema relational constraints. Direct consumers to move are Concelier projection/read services, Policy’s PostgresFindingsLookup, the CLI compact-vuln source, and mirror rebuild/export code.
Quantified causes
1. One source fact is represented repeatedly
vex.claims 3,535,816
vexhub.statements 3,535,816
vuln evidence kind vex-claim 3,535,816
vuln evidence kind vexhub-statement 3,535,816
vuln VEX observations 7,071,632
PostgresVexHubProjectionSink.BuildStatementRow derives VexHub fields from the claim: provider, document digest, vulnerability, product, status, justification, detail, and timestamps. Concelier then reads both and creates two full observations and two evidence references.
The coarse join (provider/source, document, vulnerability, product) returns 3,535,816 apparent pairs, but exact source-statement lineage exposes current damage:
claims missing their exact derived statement 424
statements without their exact originating claim 424
exact-lineage pairs with a different status 68
exact-lineage justification/detail mismatches 0
The 424/424 break is consistent with retention choosing a different statement/claim inside the too-coarse group. Equal counts do not prove provenance integrity.
One observation plus one evidence reference costs about 4.3 kB using total relation bytes/live rows. The second VEX representation therefore accounts for roughly 15 GB of direct observation/evidence storage at this corpus, plus repeated index, TOAST, aggregation, and replay cost. This is an estimate, not a cleanup quota; v2 must measure its result.
2. Full sweeps rewrite unchanged rows
All three observation upserts and all evidence upserts use unconditional ON CONFLICT ... DO UPDATE. Linkset rebuild also unconditionally updates every column and sets rebuilt_at = NOW(). vuln.issue_observations has a BEFORE UPDATE timestamp trigger. A logically idempotent sweep is physically non-idempotent.
The existing full-resweep test checks only unchanged row counts. It does not assert zero row versions or WAL. In a five-minute live measurement, update counters rose by 9,952 for observations/evidence and 8,579 for linksets. From the 09:35:16 UTC statistics reset to 11:39 UTC, the tables accumulated 82,659 observation/evidence updates and 71,288 linkset updates. Database WAL was about 19.1 GB in that interval; it includes all workloads and is not attributed solely to this projector.
Required guard:
ON CONFLICT (...) DO UPDATE SET ...
WHERE target.content_hash IS DISTINCT FROM excluded.content_hash
OR target.is_deleted IS DISTINCT FROM excluded.is_deleted;
Linkset and gate writers need the same semantic-change rule. Rebuild timestamps belong in an audit/run record, not the hot row’s equality contract.
3. Retention deletes valid current facts and leaves derived state stale
Current retention partitions claims by (tenant, vulnerability_id, product_key, provider_id), statements by (source_id, vulnerability_id, product_key), and observations by (tenant_id, issue_key, source_kind, source_key). None includes applicability/range or stable source-statement identity.
A read-only characterization using that grouping proves the defect:
applicability range [3.0,4.0) -> row_number 1 -> kept
applicability range [1.0,2.0) -> row_number 2 -> marked for delete
Both ranges can be simultaneously valid. Age/latest is not a current-applicability rule.
The worker log shows two destructive sweeps after process starts on 2026-07-22:
05:48 UTC: deleted 31,698 vuln.issue_observations
09:42 UTC: deleted 15,616 vuln.issue_observations
The second followed a worker restart and five-minute initial delay. There is no durable last-run lease, so restarts can re-run the sweep. Because the first run’s effectively unbounded batch loop completed, the later 15,616 candidates are consistent with concurrent projection recreating rows; live insert/update counters support that inference, but the current schema has no audit record that can attribute every recreated row.
Deletes and orphan cleanup run as separate autocommit operations. The worker does not collect affected non-orphan issues, rebuild linksets/gates, and advance a durable watermark in one transaction. The 699,101 stale linksets are the live result.
4. Coordination is only partly durable
The incremental cursor is persisted, but the full-sweep watermark is an in-process dictionary. Although issue_projection_state.last_full_sweep_at exists, the cursor store neither reads nor writes it. On restart the service records now in memory and postpones a full sweep. There is no database lease/fencing token, so replicas can sweep concurrently.
Advisory ordering uses GREATEST(created_at, updated_at) and sees in-place changes. VEX claims use only created_at; VexHub statements use only ingested_at. An in-place VEX update is invisible until a full sweep, while restarts can postpone it.
5. Capacity controls miss the projection
The deployed Excititor guard is configured for an 80 GB database ceiling, 5,000 documents/run, and 250,000 claims/run. The database is already 91 GB. The guard exists per provider run and checks only during provider ingestion; it does not govern Concelier sweeps, VexHub repair, mirror rebuild, retention churn, or other paths. The Web process only displays the ceiling.
Every ingest/projection path needs a shared capacity contract.
6. Bloat and maintenance are secondary
Autovacuum is globally enabled. The large tables had no last_autovacuum/last_autoanalyze after the 09:35 UTC statistics reset. Default scale factors (vacuum 0.20, analyze 0.10) are too coarse for multi-million-row update-heavy tables. pgstattuple is not installed, and installing it would mutate the database, so exact bloat was not measured in phase 1.
The sizes are largely explained by live wide rows, JSON, TOAST, and indexes. A sample gate row averaged 2,207 inline bytes and 3,136 bytes across five JSON decisions before TOAST effects; the table also has 5.7 GB indexes and 4.0 GB TOAST. Tuning cannot remove an unnecessary live semantic copy.
Focused read-only reproductions
The phase-1 tests ran inside BEGIN ... READ ONLY transactions. They are intentionally diagnostic and must not be repurposed as cleanup SQL.
Disjoint-range characterization using the current retention partition:
WITH sample(vulnerability_id, product_key, provider_id, applicability_range, last_seen_at) AS (
VALUES
('CVE-2099-0001', 'pkg:generic/acme/widget@1', 'provider-a', '[1.0,2.0)', TIMESTAMPTZ '2026-07-21 10:00:00+00'),
('CVE-2099-0001', 'pkg:generic/acme/widget@1', 'provider-a', '[3.0,4.0)', TIMESTAMPTZ '2026-07-22 10:00:00+00')
), ranked AS (
SELECT *, row_number() OVER (
PARTITION BY vulnerability_id, product_key, provider_id
ORDER BY last_seen_at DESC) AS rn
FROM sample
)
SELECT applicability_range, rn, rn > 1 AS current_retention_marks_for_delete
FROM ranked ORDER BY rn;
Coherent linkset-integrity test:
BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY;
WITH actual AS MATERIALIZED (
SELECT tenant_id, issue_key, count(*)::bigint observation_count
FROM vuln.issue_observations GROUP BY tenant_id, issue_key
), comparison AS MATERIALIZED (
SELECT l.observation_count stored, a.observation_count actual
FROM vuln.issue_linksets l JOIN actual a USING (tenant_id, issue_key)
)
SELECT count(*) FILTER (WHERE stored <> actual) mismatched_linksets,
sum(stored - actual) net_stale_observation_excess,
max(stored - actual) max_stale_excess
FROM comparison;
COMMIT;
Exact claim-to-derived-statement lineage test. The fallback identity mirrors ResolveSourceStatementId:
BEGIN TRANSACTION READ ONLY;
WITH expected AS (
SELECT c.claim_hash, c.provider_id, c.document_digest,
c.vulnerability_id, c.product_key, c.status, c.justification, c.detail,
COALESCE(
NULLIF(md.metadata_json->>'openvex.statement.id', ''),
NULLIF(md.metadata_json->>'csaf.tracking.id', ''),
'sha256:' || encode(sha256(convert_to(
c.provider_id || '|' || c.document_digest || '|' ||
c.vulnerability_id || '|' || c.product_key, 'UTF8')), 'hex')) expected_statement_id
FROM vex.claims c
LEFT JOIN vex.claim_metadata_dedup md ON md.content_hash = c.metadata_hash
), matched AS (
SELECT e.*, s.id, s.status statement_status,
s.justification statement_justification, s.status_notes
FROM expected e
LEFT JOIN vexhub.statements s
ON s.source_id = e.provider_id
AND s.source_statement_id = e.expected_statement_id
AND s.vulnerability_id = e.vulnerability_id
AND s.product_key = e.product_key
)
SELECT count(*) FILTER (WHERE id IS NULL) claims_missing_exact_statement,
count(*) FILTER (WHERE id IS NOT NULL AND status IS DISTINCT FROM statement_status) exact_status_mismatches,
count(*) FILTER (WHERE id IS NOT NULL AND justification IS DISTINCT FROM statement_justification) exact_justification_mismatches,
count(*) FILTER (WHERE id IS NOT NULL AND detail IS DISTINCT FROM status_notes) exact_detail_mismatches
FROM matched;
COMMIT;
The physical no-op test used two snapshots of pg_stat_user_tables.n_tup_upd around a five-minute interval while source counts were stable. The v2 acceptance test must use an isolated database and a stricter per-test audit/WAL boundary rather than shared live statistics.
Target physical ownership
| Database | Owner | Contents |
|---|---|---|
stellaops_platform | Platform | Platform control-plane only; no vulnerability corpus/projection |
stellaops_concelier | Concelier | Immutable advisory sources, normalized source facts, outbox/cursors |
stellaops_excititor | Excititor | Immutable VEX documents, normalized fact revisions, verification, outbox/cursors |
stellaops_vexhub | VexHub | Public/global distribution and verification read model if retained as a durable boundary |
stellaops_vuln | Vulnerability serving plane, owned by Concelier | Current semantic facts, provenance, overlays, hot linksets/gates, leased checkpoints, separate history |
stellaops_vuln is deliberately not Platform. If architecture colocates it operationally with Excititor, it must still be a dedicated physical database/role; schema-only separation inside stellaops_platform is rejected.
flowchart LR
subgraph CDB["stellaops_concelier"]
AR["Immutable advisory documents"] --> AF["Normalized advisory facts"] --> AO["Transactional outbox"]
end
subgraph EDB["stellaops_excititor"]
VR["Immutable VEX documents"] --> VF["Normalized VEX fact revisions"] --> VO["Transactional outbox"]
end
subgraph HDB["stellaops_vexhub"]
VH["Distribution and verification view"]
end
subgraph VDB["stellaops_vuln"]
PF["Current semantic facts"]
PR["Provenance and lineage"] --> PF
TO["Tenant overlays"] --> LS["Hot issue linksets"] --> GD["Hot gate decisions"]
PF --> LS
CP["Leased checkpoints and fencing"] --> PF
PF --> HI["Partitioned history/archive"]
end
AO -->|"idempotent changes/tombstones"| CP
VO -->|"idempotent changes/tombstones"| CP
VO --> VH
VH -. "lineage/verification, not a second fact" .-> PR
GD --> POL["Policy/API/CLI readers"]
PLAT["stellaops_platform"] ~~~ VDB
Canonical identity and deduplication
Use two identities:
- Issue/linkset identity:
(tenant scope, canonical vulnerability id, canonical product identity). This coarse key aggregates facts. - Semantic fact identity:
(issuer/source, stable source_statement_id, canonical vulnerability id, canonical product identity, normalized applicability selector/range hash). Revision/content hash is a value of the stable fact, not its sole identity.
Rules:
- Disjoint applicability ranges are distinct facts even with the same issuer/vulnerability/product.
- Independent issuers remain independent facts even when text is identical.
- An Excititor claim and VexHub statement derived from it are one fact with multiple provenance/processing records. Verification, signing fingerprint, and distribution identity enrich lineage; they do not create a second observation.
- A VexHub statement not originating from Excititor remains an independent source fact.
- Raw documents are immutable/content-addressed in the source database. The serving database stores narrow references/digests, not repeated raw JSON.
- Current fact and revision/history are separate. Superseding a revision never age-deletes a still-applicable fact/range.
- Tenant overlays reference global facts/issues and contain local trust, acceptance, exceptions, or local VEX without copying global payloads.
Transactional reconciliation
- The source commits a normalized revision/tombstone and monotonically ordered outbox record in one source transaction.
- One destination projector holds a renewable lease with fencing token per stream.
- In one
stellaops_vulntransaction it applies content-guarded fact/provenance changes, tombstones removed facts, collects affected issues, rebuilds linksets/gates, and advances the checkpoint. - Failure rolls back the destination transaction/checkpoint. Replay is safe through deterministic identities/hashes.
- Full reconciliation uses a source generation plus completed marker. Only after completion may absent facts be tombstoned. Age alone never means “not applicable.”
- Removing one provenance record does not tombstone a fact while another active provenance remains. Removing the last does and rebuilds derived rows transactionally.
Phased remediation and rollback
Phase 0 — urgent safeguards; runtime approval required
- Put all current claim/statement/issue supersede deletion into dry-run or disable it until stable identity/reconciliation exist. At minimum remove Excititor’s authority to delete
vuln.issue_*; evidence shows all three keys are unsafe. - Do not purge,
VACUUM FULL, reindex, truncate, or reclaim disk. - Add read-only alarms for lineage gaps, linkset/gate drift, cursor/lease age, and database/host headroom.
Phase 1 — stop new corruption/write amplification in v1
- Excititor migration
010_vex_fact_outbox.sql: stable source fact/applicability identity, revision/tombstone, source generation, outbox. - Concelier migration
025_advisory_fact_outbox.sql: equivalent advisory contract. - VexHub migration
004_origin_lineage_identity.sql: explicit origin kind/id and source-statement-preserving uniqueness. - Add semantic change guards to observation, evidence, linkset, VexHub, and gate upserts.
- Replace process-local sweep/retention scheduling with durable leased checkpoints/fencing.
- Add a safe v1 reconciliation command that rebuilds affected linksets/gates without deleting source facts. Default audit/dry-run; mutation needs approval.
Rollback is forward-only: disable new projectors/outboxes and return readers to v1. Do not remove columns/tables or edit applied migrations.
Phase 2 — side-by-side v2
- Add a dedicated persistence assembly with
001_v1_vulnerability_projection.sqltargetingstellaops_vuln: facts, history, provenance, overlays, linksets, gates, inbox/checkpoints, leases, audit. - Add distinct database roles/connections. Remove vulnerability-store fallback to
STELLAOPS_POSTGRES_CONNECTION; fail closed on a missing owner connection. - Wire owner startup migrations; Platform must not create/migrate these schemas.
- Backfill from source owners, not stale v1 linkset/gate rows.
- Shadow Policy, API, CLI compact-vuln, and mirror reads. v1 remains authoritative during comparison.
Rollback stops v2 consumers/projectors and leaves v2 intact; v1 readers continue.
Phase 3 — validate and cut over
Required for two complete source generations plus agreed soak:
- zero valid range loss;
- complete deterministic provenance and no unexpected claim/VexHub double facts;
- zero stale linkset counts/hashes and gate inputs;
- replay produces zero logical/hot-table changes beyond bounded checkpoint/audit rows;
- tombstones rebuild facts/linksets/gates transactionally;
- Policy/API/CLI shadows match approved semantics;
- lease fencing/failover prove one writer;
- backup restore is tested;
- physical capacity gate is green.
Cutover drains to a recorded watermark, switches readers reversibly, and keeps v1 read-only for rollback. Rollback switches readers, not migrations.
Phase 4 — explicit retirement/reclaim
Only after rollback window, verified backup, signed reconciliation, and separate destructive approval:
- prove no v1 vulnerability reader/writer remains;
- retire old
vuln,vex,vexhub, andconcelierdata with an explicit DBA runbook, not startup migration; - reclaim PostgreSQL/VHDX space only after blocks are safely released.
Capacity gate
| Need | Temporary capacity |
|---|---|
| v2 physical build before measured dedup | 60–100 GB |
| WAL, indexes, sort/temp, checkpoint/replication headroom | 40–80 GB |
| verified full backup/restore artifact | about 100 GB, preferably off-host |
| contingency | 40–60 GB |
If backup is local, reserve roughly 240–340 GB additional physical capacity. With off-host backup, reserve at least 140–240 GB on the database host plus off-host backup capacity. Current C: free space is 45.3 GiB, so the gate is red despite the sparse Linux filesystem showing 711 GiB.
PostgreSQL after the model correction
Initial values to validate on v2, not apply in phase 1:
- Current fact/linkset/gate tables:
autovacuum_vacuum_scale_factor=0.01,autovacuum_vacuum_threshold=20000,autovacuum_analyze_scale_factor=0.005,autovacuum_analyze_threshold=10000,autovacuum_vacuum_insert_scale_factor=0.02. - Small high-churn lease/queue/checkpoint tables: scale factors
0, thresholds around 1,000. fillfactor85–90 only on demonstrated update-hot current tables; append-only history stays 100.- Time-partition history, not current applicability facts.
- Recreate only indexes justified by v2 plans/statistics.
- Temporarily raise WAL/checkpoint capacity for approved backfill from measured batch WAL and capacity; current
max_wal_size=1GBis too small for bulk rebuild. - Targeted
ANALYZEafter each stage; concurrent index creation where supported; blocking rewrites only on retired/offline data.
Metrics and limits for every path
Cover advisory/VEX ingest, VexHub project/repair, issue projection, mirror rebuild, reconciliation, and archive:
- rows/bytes per source, fact, provenance, overlay, linkset, gate, and history partition;
- raw-to-fact, fact-to-provenance/linkset, and attempted-to-actual write amplification;
- no-op candidates versus inserts/updates/deletes and WAL per batch;
- outbox lag, checkpoint age, lease/fencing, retries/poison records;
- provenance gaps, duplicate identity, tombstoned-but-serving facts, linkset/gate drift;
- database/tablespace/WAL/backup/sparse-volume/physical-host headroom;
- per-source/global quotas that pause all writers before the safety floor with an operator-visible reason/recovery action.
Required tests
| Proof | Required test |
|---|---|
| No valid range lost | Two same-issuer/vulnerability/product disjoint ranges survive ingest, replay, reconciliation, and history compaction; deleting one leaves the other |
| Provenance retained | Excititor claim + derived VexHub statement yields one fact/two lineage records; external VexHub-only remains independent |
| Current break detected | Fixtures reproduce 424/424 lineage and 68 status-drift classes and fail until repaired/tombstoned |
| Deletes reconcile | Remove one provenance, last provenance, and one of several ranges; assert exact fact/linkset/gate in one transaction; injected failure rolls back checkpoint too |
| Replay idempotent | Second generation changes zero semantic/hot rows and only bounded checkpoint/audit state |
| No-op avoids writes | PostgreSQL stats/WAL or trigger audit proves zero hot-table updates |
| Single writer | Two instances contend; only leased fencing token commits; expired owner cannot commit |
| Update visibility | In-place revision emits outbox and reaches v2 without full sweep; restart resumes durable checkpoint |
| Side-by-side parity | Full corpus counts, identities, ranges, provenance, linksets, gates, Policy/API/CLI, intentional-difference ledger |
| Backup/rollback | Restore fresh v2, cut over, inject failure, switch to untouched v1, verify both directions |
Existing tests are insufficient: full-resweep asserts counts, not writes; retention has no focused proof for disjoint ranges/source statements or downstream rebuild.
System-wide inventory (phase 2, 2026-07-22)
The second read-only pass widened the evidence from the vulnerability plane to the whole estate:
stellaops-postgres(postgres:18.1, one volume) holds three databases:stellaops_platform91 GB,stellaops_authority252 MB,postgres7.7 MB. Authority is the only service with its own database — the per-service pattern already works here.- ~46 service schemas share
stellaops_platform. The vulnerability plane (vuln71 GB +vex9,625 MB +vexhub5,908 MB +concelier2,322 MB) is ~89 GB = 97.8% of the database. All other services combined are < 3 GB (largest:scanner2,110 MB,advisoryai271 MB,timeline199 MB;platformitself 2,024 kB). - Every service except Authority receives the same
STELLAOPS_POSTGRES_CONNECTIONin compose; the full deployable→schema→connection matrix and the cross-service SQL register are in database-ownership-matrix.md. - Observation split measured:
vex7,071,632,advisory1,156,589,distro365,770 (total 8,593,991). - Flattening proof:
count(*),count(DISTINCT (provider_id, vulnerability_id, product_key)), andcount(DISTINCT (provider_id, document_digest, vulnerability_id, product_key))overvex.claimsare all exactly 3,535,816. The coarse retention identity has already collapsed the corpus to one claim per group, so any lost multi-statement/multi-range structure is no longer observable post-hoc. v2 backfill can stop future loss but cannot recover it; parity must compare against sources, not v1. - Sampled widths (TABLESAMPLE): observation row 1,400 B of which 938 B JSON (
facts_json614,references_json275,aliases44,conflicts_json5); gate row 1,966 B + 3,491 B across the five decision JSONs; claim 602 B; statement 543 B.vexhub.statementscarries 13 indexes — 3,993 MB of index on a 1,904 MB heap. - Additional cross-owner write found: Excititor’s
PostgresVexHubProjectionSinkwritesvexhub.statements, a schema migrated/owned byvexhub-web(StellaOps.VexHub.Persistence).
Decisions requiring approval (VDPS-D1 … VDPS-D12)
Direction revised 2026-07-22 (owner review): the target moved from four separate databases to a single consolidated
StellaOps.Vulnerabilitiesmicroservice with one database, a tenant-free hub, per-tenant gates relocated to a Policy-owned projection, and a fresh rebuild from sources with no data preservation (replacing the side-by-side backfill/parity plan this document’s phase-1 sections describe). The findings above remain valid evidence; the current decision set D1–D12, with proposed outcomes, alternatives, and consequences, lives in ADR-039 rev-2, with the target model in vulnerability-data-plane-v2-design.md.
Summary of the current set: D1 out-of-platform; D2 consolidation into StellaOps.Vulnerabilities (one service, one DB, plugins for sources and mirrors, lookup APIs); D3 per-tenant gates move to Policy, CLI export reads the hub’s global API; D4 fact/issue identity with applicability; D5 outbox/lease standard; D6 retention freeze only (no v1 repair); D7 fresh re-ingest with coverage-based gates; D8 ~100 GB dedicated vulndata volume; D9 separate destructive decommission (with recommended pre-drop snapshot); D10 tenancy/IdP owned by Authority (kills the platform-DB coupling); D11 doctor-as-plugins-per-service; D12 StellaOps.<Service>.Persistence naming standard.
Evidence pointers
- Shared DB config:
devops/compose/docker-compose.stella-services.yml(Concelier 1332; Excititor 1470/1554; VexHub 1608/1610; Policy 1750/1751). - Historical shared-DB decision:
docs/modules/concelier/architecture.md(225, 1086–1107). - Unsafe grouping/deletes:
src/Concelier/StellaOps.Excititor.Worker/Scheduling/VexRetentionSweepService.cs(18–33, 59–160, 314–340, 379–456). - Claim-to-VexHub derivation:
src/Concelier/__Libraries/StellaOps.Excititor.Persistence/Postgres/VexHub/PostgresVexHubProjectionSink.cs(461–487, 550–579). - Duplicate projection/unguarded upserts:
src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Postgres/Repositories/PostgresIssueProjectionStore.cs(433–490, 582–632, 710–761, 855–901). - Process-local sweep:
src/Concelier/StellaOps.Concelier.WebService/Services/IssueProjectionBackgroundService.cs(31–36, 87–103, 152–174). - Cursor omits sweep/lease:
src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Postgres/Repositories/PostgresIssueProjectionCursorStore.cs(23–79). - Gate queue hash trigger:
src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Migrations/024_issue_gate_decision_dirty_queue.sql(22–60). - Existing gate change guard:
src/Concelier/__Libraries/StellaOps.Concelier.Persistence/Postgres/Repositories/PostgresIssueGateDecisionProjectionStore.cs(136–159). - Shallow idempotency test:
src/Concelier/__Tests/StellaOps.Concelier.Persistence.Tests/IssueProjectionIncrementalDrainTests.cs(188–239). - Excititor-only guard:
src/Concelier/StellaOps.Excititor.Worker/Scheduling/VexIngestGuardrail.cs(25–27, 48–66, 105–166). - Direct readers:
src/Policy/StellaOps.Policy.Engine/Services/PostgresFindingsLookup.cs(1083, 1135);src/Cli/StellaOps.Cli/Commands/VulnDb/PostgresCompactVulnDbCorpusSource.cs(144–145).
