Concelier Exporters — Operations

This guide is the operational reference for the Stella Ops Concelier export surfaces — federation bundle, mirror, and delta. It is intended for operators who trigger, monitor, and tune exports, and for engineers integrating with the export APIs. For the bundle format itself, see federation-bundle-export.md. For mirror deployment, see the mirror operations guide.

Surfaces

When exposed through the Stella Ops Gateway, the Concelier-owned federation paths (/status, /exports*, /export*, /import*, /sites*) must be routed to Concelier before the broader Platform federation-management fallback for /overview, /regions, and promotion controls. Otherwise read-only export probes can return a Gateway/Platform 404 even though Concelier’s endpoints are enabled.

Async federation export

Introduced by SPRINT_20260513_002. The synchronous /export path could not keep an HTTP connection alive long enough for large tenants (~50 k canonicals hit the Postgres statement_timeout); the async surface replaces it with a job that streams the bundle to S3/RustFS and notifies the requester when it lands.

The export reader pages canonical advisories with (updated_at DESC, id ASC) keyset cursors and disables exact COUNT(*) OVER() totals on streaming pages. Preview/count endpoints keep exact counts, but bundle export page fetches favor bounded IO so Postgres can return each 500-row page from idx_advisory_canonical_keyset. The consolidated baseline 001_v1_concelier_baseline.sql drops the older single-column idx_advisory_canonical_updated index (DROP INDEX IF EXISTS at line 2644 — folded in from the pre-1.0 031_drop_advisory_canonical_updated_index.sql, now archived and not embedded) because the composite keyset index covers that lookup shape and prevents planner drift back to an incremental-sort path. The bundle writer also avoids buffering the full tar or compressed bundle in managed memory: canonical, edge, and deletion NDJSON streams are spooled to process-local temp files, then replayed into the zstd tar writer. Canonical and deletion passes suppress source-edge hydration unless source filters require it; the edge pass still writes edges.ndjson. The manifest bundleHash, async job metadata hash, and signing input use the same manifest-declared hash so import signature verification sees the identity the exporter recorded.

A deterministic bounded-working-set proof for the streaming pipeline lives in StellaOps.Concelier.Federation.Tests.Integration.FederationExportWorkingSetTests (Sprint 20260513_001 E6/E0). It drives the real BundleExportService + DeltaQueryService through a lazily-generated store that never holds the dataset itself, so any reintroduced full-load in the production path shows up directly as super-linear managed-heap growth. Measured on a dev host: a full 141 753-canonical

• 141 753-edge export completes in ~1.3 s, ~3 MB compressed, with a peak live managed-heap delta of ~4 MB and keyset-only paging (max page 500, no OFFSET); the 10 k vs 140 k peak heap is flat (~1.2 MB each), confirming peak working set is independent of dataset size and far under the 1 GB / 60 s acceptance budget. The remaining gap is a live full-DB run against a populated Concelier (wall-clock and fresh-replica row-count parity), not a source-level streaming defect.

Flow

1. Trigger — POST /api/v1/federation/exports with body { sinceCursor?, maxItems?, compressLevel?, sign? }. Requires tenant context and scope advisory:read. Returns:

   202 Accepted
   Location: /api/v1/federation/exports/{exportId}
   
   {
     "exportId": "<guid>",
     "runId":    "<guid>",
     "status":   "queued",
     "message":  "You will be notified when the bundle is ready."
   }
   

2. Run — the federation:bundle:export job (StellaOps.Concelier.Federation.Jobs.FederationBundleExportJob) streams a zstd bundle directly to the configured S3 bucket (Federation:Publish:S3:BucketName, default concelier-exports) under feedser-bundle/{tenantId}/{yyyymmdd}/{sha256}.zst, with metadata tags bundle-version, bundle-hash, tenant, since-cursor, export-cursor. Job rows live in vuln.federation_exports.

3. Notify — on terminal status the job posts concelier.federation.bundle.ready (or .failed) to Notify. The requester receives both an in-app inbox row and an email rendered from the seeded templates; both carry the export ID, bundle SHA-256, byte size, and a 24-hour pre-signed GET URL (or the failure reason). The async trigger forwards the caller’s email / ClaimTypes.Email claim to Notify as recipientEmail; if the caller token has no email claim, the in-app row is still created but the email leg is intentionally skipped until user-preference lookup is implemented.

4. Download — fetch the bundle via the signed URL. Re-mint a fresh URL any time with GET /api/v1/federation/exports/{exportId}.

Polling

curl -H "X-StellaOps-TenantId: $T" -H "Authorization: Bearer $TOKEN" \
  https://concelier.example/api/v1/federation/exports/$EXPORT_ID

Response example:

{
  "exportId":   "...",
  "status":     "succeeded",
  "bundleHash": "sha256:...",
  "byteSize":   12345678,
  "sinceCursor":  "...",
  "exportCursor": "...",
  "s3Key":      "feedser-bundle/acme/20260513/<sha256>.zst",
  "signedUrl":  "https://rustfs/...",
  "expiresAt":  "2026-05-14T12:00:00Z",
  "startedAt":  "...",
  "completedAt":"..."
}

Deprecated synchronous surface

GET /api/v1/federation/export and GET /api/v1/federation/export/preview stream the bundle (or preview stats) inside the HTTP response. They are retained for short-circuit use against small tenants only and will be removed once Export Center workers (EXPORT-SVC-37-001/002) own the surface end-to-end. Production operators must migrate to the async endpoints; the deprecation note is reflected in the OpenAPI/Scalar summary.

Tactical exporter job timeouts

export:json, export:trivy-db, and export:vuln-db-compact are still tactical in-Concelier exporters while Export Center takes over the durable bundle surface. Their built-in scheduler timeout is 90 minutes with a 10-minute lease. The previous 10-minute JSON timeout cancelled a seeded local export at 09:59 after the DI activation fix started the job successfully; the previous 20-minute Trivy DB timeout cancelled large seeded exports at 19:59 before the exporter could finish. The timeout lift prevents premature cancellation, but it does not close the streaming/memory work for every exporter. JSON export now streams advisories through repository-backed pagination, falls back to stored advisories when an event-log replay is unavailable, and deduplicates replayed canonicals that map to the same output path. The 2026-05-16 seeded live proof exported 9,131 advisories in 2m29s with digest sha256:a5b358400b00dd2e2581b7008fd03c7df6736295192d9be1baa193d25573a506. The Trivy DB exporter’s materialization step is JsonExportSnapshotBuilder, which now streams advisories from IAdvisoryStore.StreamAsync and, on the non-mirror path (retainAdvisories: false), writes each advisory to disk and releases it instead of buffering the dataset. The bounded-working-set proof is StellaOps.Concelier.Exporter.Json.Tests.JsonExportSnapshotBuilderTests.WriteAsync_StreamsLargeFixtureWithBoundedWorkingSet (Sprint 20260513_001 E2): streaming 50 000 lazily-generated advisories keeps the snapshot builder’s peak live managed heap well under 256 MB (and far under the 1 GB acceptance), with no advisory list retained.

The downstream trivy.db (BoltDB) + metadata.json build is now produced by an in-repo clean-room Trivy-compatible BoltDB writer (TrivyDbBoltBuilder + Bolt/BoltDbWriter) with no external trivy-db builder dependency. The schema derivation and clean-room boundary are documented in docs/modules/concelier/trivy-db-cleanroom-design.md. A black-box round-trip test (TrivyDbBoltBuilderRoundTripTests) writes the DB, parses it with an independent BoltDB reader, and asserts a known advisory round-trips (write -> read -> same FixedVersion/Severity/DataSource).

The compact vuln-DB exporter (StellaOps.Concelier.Exporter.VulnDbCompact) is an optional tactical runtime plugin for server-side production of the Stella-native matching projection used by stella sbom check. It streams IAdvisoryStore.StreamAsync into CompactVulnDbWriter, writes vuln-db.sqlite, manifest.json, optional vuln-db-by-sa.sqlite, and a deterministic vuln-db.tar.zst bundle, and registers scheduler job kind export:vuln-db-compact with the same 90-minute timeout and 10-minute lease as the other tactical exporters. The direct stella vuln-db export CLI path remains the default operator producer for connected-box to air-gap distribution; the Concelier exporter exists for in-service generation and pluginized runtime proofs. Its regression tests assert reader round-trip, LocalAdvisoryMatcher matches, byte-stable bundle digest, plugin catalog discovery, runtime inventory presence, and StellaOps.Concelier.Exporter.*.dll host discovery.

In local compose, Trivy DB tactical output is explicitly rooted under the concelier-jobs volume:

• concelier:exporters:trivyDb:OutputRoot -> /var/lib/concelier/jobs/trivy
• concelier:exporters:trivyDb:Json:OutputRoot -> /var/lib/concelier/jobs/trivy/tree

The concelier-jobs-init service creates and owns these directories before concelier starts. Do not rely on relative exporter defaults in hardened containers: the process working directory is /app, which is intentionally read-only.

The Trivy DB job no longer requires any external trivy-db builder binary - the clean-room writer emits a valid BoltDB trivy.db directly from the canonical advisory store. The legacy concelier:exporters:trivyDb:Builder:* options remain in the options model for backward compatibility but are unused by the clean-room builder. EnsureAvailableAsync is unconditional (no runtime dependency to probe).

Runtime plugin binding

export:json and export:trivy-db are base Concelier exporter bundles in the runtime plugin inventory. export:vuln-db-compact is optional and therefore not required in the base bundle set, but it uses the same StellaOps.Concelier.Exporter.*.dll discovery path when mounted. In pluginized mode the Concelier host scans the read-only exporter payloads under the configured Concelier plugin directory, loads StellaOps.Concelier.Exporter.Json.dll, StellaOps.Concelier.Exporter.TrivyDb.dll, and any mounted optional exporter payloads, and binds the scheduler definitions from the mounted plugin catalog before using the local dependency context as a test/local-development fallback.

When Concelier:Plugins:RequireBaseBundleSet=true, the required base set includes both the Concelier advisory-source bundles and the two exporter bundles. Missing required exporter job types fail closed during scheduler options validation with the expected assembly, payload DLL, and Concelier:Plugins:RequiredBundles remediation text. Optional or unmounted non-base bundles remain visible as not mounted rather than being silently treated as runnable.

This binding proof is static/test evidence only. Archive readiness for live runtime acceptance still requires a rebuilt mounted-plugin stack and real operator-path smoke: trigger POST /jobs/export:json and POST /jobs/export:trivy-db, verify the job records and generated output tree, and capture the plugin probe/image audit alongside the export result.

Source-license controls. Each emitted record carries its source + license and is filtered by SourceLicenseAllowlist: CC0/MIT/BSD/Apache/CC-BY/public-domain sources are included with notices; unknown/restrictive sources are reduced to uncopyrightable facts; Ubuntu / CC-BY-SA-4.0 share-alike data is segregated into a separate trivy-by-sa.db shard and never folded into the main BUSL DB. The produced DB is a data artifact carrying the composite notice in third-party-licenses/DATA-NOTICES.md, not licensed as BUSL.

Export Center alignment

Concelier exporters are tactical implementations, not a separate strategic export product. Keep them lift-and-shift compatible with Export Center profile semantics:

The controlling decision is ADR-003: Concelier Export Center Air-Gap Alignment. Do not add Concelier-only profile names for JSON or Trivy exporters; if a tactical option cannot map to an Export Center profile field, record the gap before changing the contract.

Compose resource defaults

The default compose Concelier service is sized with the export-friendly tier (4G memory, 2 CPU). That is the baseline for seeded JSON exports and routine federation bundle exports after the streaming fixes. The docker-compose.concelier-export-mem.override.yml overlay remains available for bounded full seeded evidence runs, raising Concelier to 8G / 2 CPU while the Trivy DB exporter still has open streaming/memory work. Operators should prefer the compose resource limit as the default memory boundary; do not add a separate .NET GC hard limit unless a deployment has measured live evidence and records the tuned value with the export run.

Local development email

The dev compose stack can route federation email through the Mailpit overlay:

cd devops/compose
docker compose -f docker-compose.stella-ops.yml \
  -f docker-compose.concelier-export-mem.override.yml \
  -f docker-compose.concelier-federation.override.yml \
  -f docker-compose.concelier-federation-s3.override.yml \
  -f docker-compose.airgap-seed.override.yml \
  -f docker-compose.notifier-mailpit.override.yml \
  -f docker-compose.airgap-e2e.override.yml \
  --profile mailpit up -d

Rendered messages land in Mailpit at http://127.0.0.1:8025. The Web harness src/Web/StellaOps.Web/scripts/run-airgap-federation-e2e.mjs drives this chain, triggers a --max-items 10 export through the CLI, verifies the bundle hash, imports the bundle into the concelier-replica service backed by the isolated postgres-replica container, checks Mailpit, and asserts the live /notifications Download CTA. The replica isolation is database-level rather than schema-level because Concelier’s current SQL contract owns the canonical vuln schema and several repositories still use that qualified schema name directly. For the local 127.*/stella-ops.local stack, the harness enables Authority’s dev/CI seed migration when no explicit STELLAOPS_AUTHORITY_PASSWORD, STELLAOPS_FRONTDOOR_PASSWORD, or STELLAOPS_ADMIN_PASS is present, so fresh compose volumes converge to the documented admin dev identity. Set STELLAOPS_AIRGAP_E2E_REQUIRE_PASSWORD_ENV=1 to require an explicit credential source. Before starting compose, the harness also verifies the required local stellaops/*:dev images (console, router-gateway, platform, authority, concelier, notify-web, notifier-worker) so a missing image is reported as an actionable local build step instead of an unexpected registry pull. If the local stellaops / stellaops_frontdoor Docker networks already exist without Compose ownership labels, the harness automatically layers docker-compose.existing-networks.override.yml; set STELLAOPS_AIRGAP_E2E_USE_EXISTING_NETWORKS=0 to force compose-managed networks. The overlay disables Router registration for notify-web and the isolated concelier-replica, because this harness exercises their direct HTTP surfaces and does not require router messaging plug-ins in those two containers. The browser still reaches the inbox through the frontdoor: Router Gateway has an explicit direct reverse-proxy route for /api/v2/notify* to notify-web:8080, ahead of the generic /api catch-all. The e2e overlay also sets Federation:Publish:S3:PublicDownloadBaseUrl to the host-reachable RustFS / SeaweedFS path (http://127.1.1.3:8333/concelier-exports) and Federation:Publish:S3:UseSignedHttpRequests=true. Those options are for trusted dev/e2e S3-compatible emulators only: uploads and metadata checks still go through the S3 HTTP surface, but Concelier uses its minimal path-style Signature V4 compatibility path because local SeaweedFS/RustFS rejects the AWSSDK PUT and pre-sign wire shape even when filer-backed IAM is configured. Production AWS/MinIO deployments should leave the compatibility flag unset and use the normal AWSSDK Signature V4 PUT plus pre-signed GET path. The shared dev SeaweedFS service mounts a baseline devops/compose/seaweedfs-s3.config.json with the intended stellaops / stellaops admin identity and devops/compose/seaweedfs-security.toml with the local filer JWT keys SeaweedFS requires before it will honor S3 IAM. Keep that config in sync with CONCELIER_FEDERATION__PUBLISH__S3__ACCESSKEYID and CONCELIER_FEDERATION__PUBLISH__S3__SECRETACCESSKEY. SeaweedFS weed server -s3 still persists IAM state through its filer, so a live signed-URL proof must verify s3.config.show sees the imported user before running the F2 publisher harness; relying only on AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY is not sufficient when SeaweedFS has filer-side IAM config. The dev and infra compose definitions also start SeaweedFS with -volume.max=64. Keep that explicit setting: the default eight-volume ceiling is easy to exhaust during repeated federation-export proof runs and produces No writable volumes and no free volumes left even when S3 IAM and request signing are otherwise configured correctly. The Authority standard-plugin config grants the local human CLI client the email and advisory:ingest scopes so the password-grant token can carry the seeded admin email claim into the federation event and perform the dry-run/replica import leg. Without email, the bundle can export but the Mailpit assertion has no recipient to verify; without advisory:ingest, feedser bundle import correctly fails 403 Forbidden because the import endpoint uses Concelier.Advisories.Ingest. Normal harness teardown runs docker compose down --volumes so stale Authority seed data cannot mask changed client scopes between e2e runs; set STELLAOPS_AIRGAP_E2E_SKIP_TEARDOWN=1 only when preserving the stack for debugging. When Authority is enabled outside Development/Testing, Concelier registers the shared messaging-backed token cache before attaching bearer auth to the Notifier producer client. This keeps the service-to-service token path explicit and avoids the auth client’s fail-closed in-memory cache guard in production-like compose runs. The local federation compose overlay uses the seeded stellaops-cli-automation confidential client for that producer path; real deployments must replace it with an environment-specific Concelier service credential.