Runbook — Concelier / Excititor connector operations

Applies to: advisory/VEX feed connectors on a running stack — why a connector “does nothing”, profile drift, fixture vs real feeds, canonical ingest, and re-mapping without re-fetch. Companion: docs/agent-playbooks/concelier-connector.md (field summary), docs/runbooks/plugin-cross-alc-missingmethod.md (the DLL/MissingMethodException class), docs/modules/concelier/** (architecture & per-connector ops). Verify against src/ and the live /jobs/definitions + /internal/plugins/status — these drift.

Connectors ship in profiles. base ≈ 13 sources; recommended ≈ 38 and adds metasploit/exploitdb/cisco/msrc/etc. A base-only image silently drops the extra bundles:

Fix = redeploy with the recommended bundles (no code change). Always check which profile the running image carries before treating an empty source as a bug.

Fixture-mode vs real feeds (the other “does nothing”)

The default running Concelier is fixture-mode — every connector points at the dead advisory-fixture.stella-ops.local. A “realfeeds” override sends only ~10 curated sources to the real internet; many sources (e.g. kev/auscert) are not in it.

A 404 on POST /jobs/source:<key>:fetch therefore has several distinct causes — distinguish before filing a bug:

Canonical ingest & the host bridge

The PostgresAdvisoryStore host bridge makes all connectors canonical-ingest, fixing legacy-only sources that previously showed 0 advisories. Source taxonomy is Group A/B/C (see the host-canonical-bridge notes / module docs). The matcher historically read legacy advisory_affected, not the canonical store — verify which store a matching/consensus query reads before trusting its output.

Re-map existing edges WITHOUT a re-fetch

To re-derive vendor_status (or re-run a mapper) on already-fetched docs:

  1. Inject the doc IDs into vuln.source_states.cursor -> pendingMappings.
  2. POST source:<key>:map.

The mapper writes with ON CONFLICT DO UPDATE, so it overwrites in place. The real gate for whether work happens is an empty pendingMappings(not hash dedup) — if it’s empty, nothing maps. The edge upsert keys on (canonical_id, source_id, source_doc_hash), and source_doc_hash excludes vendor_status, so re-mapping an unchanged doc is stable and re-derives status in place.

Practical details that trip people up:

When map “succeeds” but changes 0 edges

That’s the swallowed cross-ALC / host-skew failure, not success — a mapper hit a MissingMethodException (Advisory..ctor, or a NuGet.Versioning/JsonSchema.Net doubly-loaded type) which was caught and logged as success while the doc went failed. Resolve via docs/runbooks/plugin-cross-alc-missingmethod.md (prune + coherent rebuild), then re-map and confirm real edge movement.

Verify (live forcing-function)

A real fetch → parse → map producing changed source_states / advisory edges (and a doc moving failed → mapped) is the proof. “Job succeeded” with no data movement is the failure signature, not a pass.

Reference-based connector credentials (kill plaintext feed creds)

Custody review P0 #2 (SPRINT_20260703_003). Connector secrets — GitHub PAT, Red Hat subscription token — used to live plaintext in vuln.sources.config JSONB. They now support secret references resolved through the unified ISecretProvider (ADR-031/032: builtin | vault | openbao), so nothing sensitive is stored in the sources table at rest.

Applies to single-value connectors — ghsa (apiToken), redhat (apiToken/subscriptionToken), redhat-csaf (subscriptionToken) — and, since the 2026-07-04 composed-credential follow-up, to composed-credential connectors: cisco/microsoft (OAuth clientId+clientSecret) and cve (apiOrg/apiUser/apiKey). Each composed field can independently hold a reference (typically only the secret half: clientSecret / apiKey); the resolver resolves the reference-bearing fields and composes the pair/triple the connector’s auth expects. An unresolvable reference for any half fails closed — the connector refuses to send.

How it resolves (in-host, not the Platform HTTP bridge)

Concelier owns its own Postgres + secret provider, so it resolves references in-host via SecretProviderRemoteCredentialResolver (chosen over AddPlatformRemoteCredentialResolver so a feed pull has no runtime dependency on the Platform WebService — air-gap posture). Every credentialed source fetch job now invokes ConnectorCredentialFetchWarmup before it reads connector options or creates the HTTP client. The warm-up uses the envelope-bound tenant when present and the source storage DefaultTenant for background cron jobs, resolves the connector’s secret-bearing config field through ISecretProvider, caches the plaintext (30 s TTL), and invalidates typed/HTTP-client option caches so even an early options read is rebuilt. The synchronous connector overlay (IPostConfigureOptions) then picks it up via TryGetCached on the first real pull. A plain literal (no scheme) is left as-is (back-compat).

Fail-closed: a recognised-but-unresolvable reference throws RemoteCredentialResolutionException; the pre-fetch warm-up propagates that exception and the connector is not invoked. The overlay’s secret step additionally refuses to copy a reference-shaped value verbatim, so direct/non-job option reads also never send the pointer as a credential.

Set a reference (operator)

Store the reference in the connector’s secret field via the source-configuration surface (same control as a plaintext token today), e.g.:

# builtin (sealed in Concelier's own crypto.secret_store):
stella db connectors configure ghsa --set apiToken='builtin://concelier-connector/ghsa/apiToken'
# external Vault / OpenBao (path-addressed, tenant-agnostic):
stella db connectors configure redhat --set apiToken='vault://secret/concelier/redhat-token'

Reference grammar: docs/implplan/specs/secret-reference-url-scheme.md.

Realfeeds override (GHSA PAT via reference)

The realfeeds override that gives GHSA a PAT should now carry a reference, not the raw ghp_…: set apiToken=builtin://concelier-connector/ghsa/apiToken (after the seal migration below) or a vault://… reference. Anonymous GHSA still works with no token.

Seal-at-rest migration + sweep

ConnectorCredentialSealMigration (a forward-only, idempotent startup task) walks every persisted single-value source, seals each field still holding plaintext into the builtin store, rewrites the field to the resulting builtin://… reference, and runs a sweep that proves no plaintext secret remains. Re-running is a no-op (an already-referenced field is skipped). For composed-credential connectors only the SECRET half is sealed (Cisco/MSRC clientSecret, CVE apiKey); the non-secret identifiers (clientId/tenantId/apiOrg/apiUser) stay plain and the resolver composes the credential at resolve time, so their auth is not broken.

Host wiring (deploy step)

The in-host resolver + seal migration require an ISecretProvider in the Concelier WebService. The host registration mirrors ReleaseOrchestrator / Notifier (SPRINT_20260703_002):

// 1) crypto prerequisites (bare NpgsqlDataSource for crypto.secret_store, AEAD, KEK sources)
builder.Services.TryAddSingleton<NpgsqlDataSource>(_ =>
    new NpgsqlDataSourceBuilder(concelierOptions.PostgresStorage.ConnectionString).Build());
builder.Services.TryAddSingleton<IAeadAlgorithm>(_ => DefaultAeadAlgorithm.Instance);
builder.Services.AddSecretProviderKeySources(builder.Configuration,
    new[] { "Concelier:SecretProtectionKey", "STELLAOPS_SECRETS_ENCRYPTION_KEY", "STELLAOPS_BOOTSTRAP_KEY" },
    registerVault: s => s.AddVaultKekSource(builder.Configuration));
// 2) unified secret provider (builtin | vault | openbao) + durable builtin store
builder.Services.AddRoutingSecretProvider(builder.Configuration);
SecretStoreServiceCollectionExtensions.AddDurableBuiltinSecretStore(builder.Services, "Concelier");
// 3) in-host connector credential resolution + seal-at-rest migration
builder.Services.AddSecretProviderRemoteCredentialResolver();
builder.Services.AddConnectorCredentialSealMigration();

Requires project references StellaOps.Cryptography.CredentialStore.Vault + StellaOps.Cryptography.CredentialStore.Persistence on the WebService. Verify at deploy: the crypto.secret_store table auto-migrates, the seal-migration log shows sweep clean, and a GHSA fetch authenticates with the resolved PAT (a live forcing-function).