Connector aggregation state

Snapshot base: 2026-05-08 04:55 UTC from the live stellaops_platform database and running Concelier/Excititor containers. CERT-FR was refreshed at 2026-05-08 06:53 UTC, and GHSA was refreshed at 2026-05-08 07:11 UTC after the public REST fix and redeploy.

Latest live recheck: 2026-05-09 10:53 UTC after the Concelier stale job-run reconciliation fix, the Excititor stale provider-heartbeat reconciliation fix, Concelier image sha256:c220f985c6957c11d1254b71de4090ba3d9673786954144f58ba5642777b06e3, and Excititor worker image sha256:ced423d06d03ffb4fe0b47fe4de0e3531e0a7c286f9e8b7d4785bbe894dce217.

• Root cause of stale Running states: persisted job/provider state recorded Running, then host cancellation or container replacement could interrupt the terminal write. Concelier had no startup/loop reconciliation from vuln.job_runs to live vuln.job_leases, and Excititor had no startup reconciliation for provider rows that still heartbeated Running after the worker exited.
• Concelier is healthy and ready after the stale-run redeploy. Startup logs show the scheduler reconciled 204 stale active job runs, then two more orphaned rows on later loops. GET /jobs/active at 10:53 UTC showed source:redhat:map, source:osv:map, and source:cert-fr:fetch still running with live leases, so they are active work, not stale rows.
• Current update: legacy arm source rows were removed in Sprint 20260601_001 because the connector failed closed and no approved machine-readable Arm feed exists. Historical counts below predate that removal.
• Excititor worker is healthy after redeploy. Startup logs show it reconciled 1 stale provider heartbeat before scheduling runs. Red Hat and Ubuntu VEX retries restarted on schedule and are now heartbeating Running as real progress, not stale rows.
• Advisory/distro source totals now show 55 source rows, 32 enabled rows, 46 rows with raw docs or advisory items, 9776 source documents, 8659 DTOs, and 44512 advisory rows.
• Raw-document anomaly repair is complete for the investigated direct-upsert rows: CVE, AWS, GCP, AMD, Intel, Siemens, MITRE ATT&CK, MITRE D3FEND, Astra, and Ubuntu all have non-empty retained source payloads; zero-byte source payload count for those rows is 0.
• CVE is no longer credential-blocked: public cvelistV5 fetch retained 1000 raw CVE JSON documents, parse produced 1000 DTOs, and map produced 999 canonical advisory rows.
• GHSA is enabled and progressing through the public REST path: 124 raw docs, 120 DTOs, and 118 advisory rows at this recheck. The optional token remains a scale/rate-limit accelerator, not a hard credential requirement.
• Apple is no longer blocked by the inherited DateTimeOffset.MinValue cursor state; live fetch, parse, and map runs succeeded and Apple now has 24 source documents, 24 DTOs, and 24 advisory rows.
• Adobe is no longer success-empty, but remains failed-live: source:adobe:fetch records All Adobe index pages failed (1 attempted) because https://helpx.adobe.com/security/security-bulletin.html times out from the container at the configured 20 second client timeout.
• Cert-IN remains failed-live after redeploy: source:cert-in:fetch fails during TLS handshake with Received an unexpected EOF or 0 bytes from the transport stream.
• VEX status at 2026-05-09 10:53 UTC: Cisco, MSRC, Oracle, Red Hat, and Ubuntu have raw VEX documents; OCI OpenVEX remains operator-configuration blocked; SUSE Rancher remains source-discovery/offline-snapshot blocked; Red Hat and Ubuntu stale Running heartbeats were moved to terminal cancelled/failed state before their new scheduled retries started.
• Exact VEX counts at this recheck: vex.vex_raw_documents = 43163; vexhub.statements = 24268422; verified VexHub statements = 42554.
• Seed export is still not ready unless performed as an explicit checkpointed export: Concelier has active source jobs with live leases and Excititor Red Hat/Ubuntu retries are running.

This page is the Stella Ops Mirror connector state record for the current bootstrap dataset. It separates rows that are actually aggregating from rows that are blocked by operator configuration, blocked by policy or source defects, or intentionally represented by a parent aggregation.

Summary

VEX trust split:

• Trusted VEX mirror data: excititor:redhat and the verified subset of excititor:cisco.
• Cache-only or unsigned/internal VEX data: excititor:ubuntu, excititor:msrc, excititor:oracle, and the non-verified Cisco historical subset.
• Configuration-blocked VEX: excititor:oci-openvex.
• Non-configuration VEX blockers: excititor:suse-rancher. excititor:oracle is aggregating only as degraded/cache-only data and remains excluded from trusted VEX publication unless a signed-trust policy is supplied.

Current completion gate:

• Concelier still has active aggregation work. /jobs/active returned source:redhat:map, source:osv:map, and source:cert-fr:fetch at the 2026-05-09 10:53 UTC recheck, and all had live leases.
• Excititor has no stale provider heartbeat at this recheck. Red Hat and Ubuntu have real scheduled retries heartbeating Running.
• Result: the current aggregation is not ready for connector re-setup testing yet. Backups can be taken, but seed export should wait until active Concelier jobs stop or are intentionally checkpointed.

Latest recheck after backup:

• Rechecked at 2026-05-08T05:36:32Z.
• Concelier still has active aggregation work: /jobs/active returned source:osv:map, started at 2026-05-08T05:30:00Z.
• Excititor still has active VEX work: excititor:ubuntu is heartbeating Running at 2026-05-08 05:36:09Z.
• Ubuntu VEX advanced from the snapshot to 16908 raw VEX docs and 10225982 VexHub statements.
• Result remains: not ready for connector re-setup testing.

GHSA redeploy and live recheck:

• Rechecked at 2026-05-08T06:48:52Z after redeploying stellaops/concelier:dev image sha256:b7c13891646c7e76d16c60b997f7fe1342f5c7fb51ea31b572b61b4a0b6a8ce8.
• GHSA is no longer credential-blocked: the source row is enabled and uses the public GitHub REST /advisories endpoint. The token remains optional for higher rate limits or enterprise routing.
• The first live fetch failed fast in 00:00:00.6900120 because GitHub returned 403 / anonymous REST rate limit exhausted for egress IP 195.149.251.242; this was an operational quota blocker, not a missing-credentials requirement.
• Rebuilt again with the detail-rate-limit cursor preservation fix and messaging transport plugin staging; final healthy image is sha256:0c0bab9c496242b02c4b604c4761abce35c776819cd3d164c074dbea341392d6.
• Live repair seeded the 50 already-downloaded GHSA raw document ids back into the cursor; manual parse 69eb85ca-1a92-f572-0d5b-bba7f41eed5b and map 7dce418a-62d2-c1d4-41f9-9c0a811138e8 succeeded.
• After quota reset, container-side https://api.github.com/rate_limit returned x-ratelimit-remaining=60; manual fetch 88e4e220-34cf-d333-58f1-bb2a7c62fd76 and scheduled fetch a9fc14d7-2391-f88f-c1c7-9ed98c348885 both succeeded.
• Final GHSA recheck at 2026-05-08T07:11:43Z: 60 source documents, 60 DTOs, and 60 canonical vuln.advisories; cursor is clean for pending docs/mappings and is progressing through the 30-day window at nextPage=3.
• A token is still recommended for production bootstrap scale. Without a token, keep the default small batch (pageSize=10, maxPagesPerFetch=1) so anonymous REST quota resets can sustain progress.

Failure split

Configuration or operator-input blocked rows:

Non-configuration blockers:

Expected empty or parent-backed rows, not failures:

• epss writes EPSS scores as an overlay, so zero vuln.advisories rows is expected.
• osv is the parent ingest for OSV documents; ecosystem item rows are attributed to buckets such as npm, pypi, go, chainguard, and wolfi.
• OSV-backed disabled buckets with item rows stay disabled as standalone schedules until an independent connector is added.

Advisory aggregations

Items are rows in vuln.advisories. Source docs and DTOs come from concelier.source_documents and concelier.dtos.

Distro aggregations

VEX aggregations

Items are vexhub.statements. Raw docs are rows in vex.vex_raw_documents.

Configuration surface cleanup

Only sources that are actually fixable by operator configuration should appear as configuration choices.

Keep configuration surfaces for:

• Advisory ghsa, cisco, and microsoft. GHSA configuration is optional and exists for higher GitHub rate limits or enterprise routing; Cisco and Microsoft remain credential/selected-mode choices for the legacy advisory sources.
• Advisory chromium, oracle, and adobe where the surface is an endpoint/mirror override and not the current blocker. Adobe should stop being empty only after the verified local fix is deployed and the live jobs rerun.
• VEX excititor:cisco, excititor:ubuntu, excititor:msrc, and excititor:oci-openvex.

Remove or suppress configuration surfaces for:

• VEX excititor:oracle, because missing public detached signatures/provider trust material is not a setup-time credential issue.
• VEX excititor:suse-rancher, because the current blocker is source/discovery shape, not a credential field operators can safely fill.
• Any future source whose live blocker is parser/source implementation, policy/export-control, or no official public mirror rather than a missing operator secret or endpoint override.

Seed export/import format

The export/import format is a Stella Ops Mirror seed archive, not a plain database dump. Format name: stellaops-mirror-seed-v1.

Archive layout:

stellaops-mirror-seed-v1/
  manifest.json
  data/
    vuln.sources.copy.zst
    vuln.source_states.copy.zst
    vuln.feed_snapshots.copy.zst
    vuln.advisory_snapshots.copy.zst
    vuln.advisories.copy.zst
    vuln.advisory_aliases.copy.zst
    vuln.advisory_cvss.copy.zst
    vuln.advisory_affected.copy.zst
    vuln.advisory_references.copy.zst
    vuln.advisory_credits.copy.zst
    vuln.advisory_weaknesses.copy.zst
    vuln.kev_flags.copy.zst
    vuln.advisory_canonical.copy.zst
    vuln.advisory_source_payload.copy.zst
    vuln.advisory_source_edge.copy.zst
    vuln.issue_observations.copy.zst
    vuln.issue_linksets.copy.zst
    vuln.issue_evidence_refs.copy.zst
    concelier.source_documents.copy.zst
    concelier.dtos.copy.zst
    vex.providers.copy.zst
    vex.connector_states.copy.zst
    vex.vex_raw_documents.copy.zst
    vex.vex_raw_blobs.copy.zst
    vex.vex_raw_attachments.copy.zst
    vex.linksets.copy.zst
    vex.linkset_observations.copy.zst
    vex.linkset_disagreements.copy.zst
    vex.linkset_mutations.copy.zst
    vex.claims.copy.zst
    vex.deltas.copy.zst
    vex.checkpoint_states.copy.zst
    vex.checkpoint_mutations.copy.zst
    vex.attestations.copy.zst
    vexhub.sources.copy.zst
    vexhub.statements.copy.zst
    vexhub.provenance.copy.zst
    vexhub.conflicts.copy.zst

Manifest requirements:

• formatVersion, createdAt, tenantId, checkpointMode, sourceDatabase, Stella Ops build/git revision when available, and DB migration watermark.
• Per-source/provider summary with key, type, enabled state, status, failure classification, trust classification, item count, first aggregation time, last aggregation time, and last error.
• Per-table row count, path, byte count, SHA-256, exported column list, optional export filter, and optional skip reason for schemas/tables not present on a deployment.
• Top-level warnings for checkpointed exports and clean-seed exclusions, including non-mapped raw source documents that were deliberately left out of concelier.source_documents/concelier.dtos.
• Active-work summary when the export is checkpointed: Concelier vuln.job_runs, Excititor connector heartbeat state, and VexHub ingestion jobs.
• Sanitized connector/source configuration. The exporter removes credential-like JSON keys and disables exported rows that are credential-blocked.
• Content hashes are carried in manifest.json; v1 does not write a separate checksum file.
• Explicit exclusion list for non-exportable secrets. Do not export provider API tokens, client secrets, registry passwords, host paths, or private trust material.
• Unified issue projection coverage: count exported vuln.issue_linksets, vuln.issue_observations, and vuln.issue_evidence_refs; link to the migration watermark that produced them.

Exporter behavior:

1. Refuse export while active work exists unless allowCheckpointedActiveJobs=true, in which case the active work is listed in manifest.json.
2. Run a read-only transaction at repeatable-read isolation.
3. Export table data through PostgreSQL COPY to Zstandard-compressed CSV chunks, preserving tenant/source IDs.
4. Export only mapped raw source documents and DTOs for clean setup replay. concelier.source_documents is filtered to status = 'mapped'; concelier.dtos is filtered to DTOs whose document is mapped. Failed, pending-parse, and pending-map raw work is operational residue, not completed aggregate data, and is reported through manifest.json warnings.
5. Export advisory source payloads through vuln.advisory_source_payload when present. Legacy archives that still carry vuln.advisory_source_edge.raw_payload are imported through a staging transform that writes one payload row per source_doc_hash and keeps edge raw_payload null.
6. Hash every chunk before writing manifest.json.
7. For space-constrained live exports, skipVexHubProjection=true omits vexhub.statements, vexhub.provenance, and vexhub.conflicts with manifest skip reasons. Full/compact seeds retain vex.claims, and the VexHub projection is rebuilt locally from that retained serving tier after import. A non-empty VEX source-only export fails closed: re-normalizing raw vendor documents can expand them into hundreds or thousands of product-grain rows and recreate the storage incident.

Importer/setup behavior:

1. During setup, enable Mirror:SeedImport so Concelier imports the seed before mirror export and source scheduler hosted services start.
2. Validate migration watermark and schema compatibility before copying data.
3. Restore successful public/no-credential rows and blocked-state explanations.
4. Do not import secret values; leave credential-required sources disabled or configuration-blocked.
5. Resume schedules from imported cursors/checkpoints so the next setup fetches only deltas.
6. Disable client command timeout and PostgreSQL statement_timeout for the archive import/export session because full setup seeds can contain multi-GB COPY chunks. Host shutdown or tool cancellation remains the cancellation boundary.
7. Accept older advisory_cvss chunks that predate the cve_id column. New exports include cve_id; older chunks import through a conflict-tolerant staging path so duplicate legacy CVSS rows do not block first setup.
8. Accept older vex.claims chunks that predate compact document columns. Legacy document_json values are staged, split into document_format, document_source_uri, document_revision, and document_signature_json, and imported with document_json set to null.
9. Destructive local clean-replay runs pause stellaops-excititor-web, stellaops-excititor-worker, and stellaops-vexhub-web while Concelier imports the seed, then restart those writers after the Concelier import marker. This keeps VEX checkpoint writers from contending with the first-load claims import.

Startup import configuration:

• Mirror:SeedImport:Enabled: opt-in startup gate. Defaults to false.
• Mirror:SeedImport:SeedDirectory: path to the stellaops-mirror-seed-v1 archive root.
• Mirror:SeedImport:TenantId: target tenant, default default.
• Mirror:SeedImport:ReplaceExisting: destructive replacement mode for controlled fresh setup tests only.
• Mirror:SeedImport:ValidateOnly: validate manifest/schema/hash without copying rows.
• Mirror:SeedImport:SkipVexHubProjection: skips optional vexhub.* rows during import. The archive must retain vex.claims or vex.vex_raw_documents; otherwise export fails closed. After VexHub migrates its schema, the Excititor Worker repairs the projection locally even when every provider schedule is disabled.
• Mirror:SeedImport:SkipWhenTargetNotEmpty: default true; lets normal restarts continue after the seed is already present.
• Mirror:SeedImport:FailStartupOnError: default true; validation/import failures stop the host before schedulers run.
• Mirror:SeedImport:DeltaApply: opt in only when SeedDirectory is a delta bundle. The startup gate verifies the locally recorded baseline digest before any write, then UPSERTs changed source rows and records the applied delta digest for the next link in the chain. Leave this false for full bundles.

A destination is not reader-ready merely because import returned success. Keep the front door out of rotation until local VEX repair reports completed=true, the enforced claim-to-VexHub anti-join is empty, and one canonical CVE/product sample is readable through Excititor, VexHub, and the Concelier issue reader. Advisory-only claims are intentionally absent from VexHub and are excluded from that parity condition.

Scoped and delta bundles:

1. Export a sealed baseline first. Use --profile source-only for the smallest re-derivable distribution bundle. A scoped baseline can repeat --source and --ecosystem; selector resolution is fail-closed, so an unknown selector aborts instead of producing an empty archive.
2. After source rows advance, export a delta with --delta-baseline <baseline>/manifest.json. The manifest records the baseline bundle digest, strict-after watermarks, and changed row counts. A no-change delta contains no data rows.
3. Import the baseline normally. This records its sealed digest in vuln.mirror_seed_state.
4. Apply the next bundle with --delta-apply. The importer compares the delta’s baseline digest with the locally recorded digest before writing, then natural-key UPSERTs changed rows and preserves unchanged rows. A missing or mismatched anchor is a hard failure.

stella mirror seed export --connection-string $dsn --output-directory C:\mirror\ubuntu-baseline --profile source-only --source ubuntu
stella mirror seed export --connection-string $dsn --output-directory C:\mirror\ubuntu-delta-001 --profile source-only --source ubuntu --delta-baseline C:\mirror\ubuntu-baseline\manifest.json
stella mirror seed import --connection-string $targetDsn --seed-directory C:\mirror\ubuntu-baseline
stella mirror seed import --connection-string $targetDsn --seed-directory C:\mirror\ubuntu-delta-001 --delta-apply

The lower-level StellaOps.Concelier.MirrorSeedTool exposes the same --profile, repeatable --source/--ecosystem, --delta-baseline, and --delta-apply contract for repository and recovery workflows. Both surfaces reuse PostgresMirrorSeedArchiveService; neither fetches from an external network.

Operational scripts:

• stella mirror seed export|validate|import|download is the operator CLI surface for creating a clean seed, validating it, copying it into the ingestion catalog, and replaying it into a local database. In source checkouts, run it through dotnet run --project src\Cli\StellaOps.Cli\StellaOps.Cli.csproj -- mirror seed <command>.
• devops/mirror/export-concelier-seed.ps1 remains the PowerShell wrapper for exporting the live database with the repository tool. Use -AllowCheckpointedActiveJobs only when the manifest should explicitly record active work. Use -SkipVexHubProjection when disk space cannot hold the duplicate VexHub projection tables.
• devops/mirror/prepare-concelier-seed-import.ps1 -SeedArchiveRoot <path> writes a compose override, env file, and setup note that mount a seed archive and enable Mirror:SeedImport for the Concelier service. The generated replay profile sets SkipVexHubProjection=true; after import, start VexHub for schema convergence and the Worker for its provider-independent local repair. Provider schedules may remain disabled.
• tools/scripts/qa/Wipe-And-Rebuild.ps1 -Confirm -PinnedSeedDir <seed> is the destructive local replay harness. For seed-import runs it prepares the Concelier override with ReplaceExisting=true, waits for the Concelier import marker, keeps VEX writers paused during import, restarts those writers afterward, and probes the frontdoor.
• devops/mirror/test-concelier-seed-import-e2e.ps1 is the repeatable bounded clean-import proof. It creates isolated source and target PostgreSQL containers, applies real Concelier and Excititor startup migrations, exports a seed with advisory and VEX issue observations, imports it through the Concelier startup gate, verifies target counts and evidence refs, and cleans up its containers on success.
• devops/mirror/seeds/README.md is the repository catalog for seed cards. The current ingestion seed card is devops/mirror/seeds/20260607T003352Z/seed-card.json; the multi-GB seed payload is stored under the seed directory through Git LFS and keyed by the manifest hash.

API surface:

• POST /api/v1/advisory-sources/mirror/seeds/export creates a seed archive at a configured or requested output path.
• POST /api/v1/advisory-sources/mirror/seeds/validate verifies manifest version, table existence, columns, byte counts, and SHA-256 without importing rows.
• POST /api/v1/advisory-sources/mirror/seeds/import imports the archive. Non-empty target tables are rejected unless replaceExisting=true.

Operational checkpoint:

• A binary pg_dump -Fc backup is useful as a safety backup, but it is not the setup seed format. The seed format must be selective, secret-free, content-hashed, and importable before schedulers start.
• Unified issue projection implementation details live in docs/modules/concelier/unified-issue-projection.md. Backfill the projection in bounded chunks before exporting the seed so SBOM/finding flows can resolve against the same skeleton after import.
• Clean setup replay is covered by devops/mirror/test-concelier-seed-import-e2e.ps1. The 2026-05-11 passing run wrote evidence to C:\temp\stellaops-mirror-seed-e2e\20260511T225811Z\e2e-result.json.
• The 2026-06-07 seed run was point-in-time validation evidence, not a current checked-in distribution payload. Its repository payload was removed during workspace cleanup. Generate a fresh sealed archive for distribution; when VexHub projection tables are skipped, the destination-local Worker repair rebuilds them from retained claims. Raw-only VEX reconstruction is intentionally refused.