Runbook: Feed Connector - NVD Connector Failures

Purpose: recover the Concelier NVD connector when the NIST CVE 2.0 feed is failing or stale — transport/endpoint problems, schema-validation quarantines, or a window cursor that needs widening. Audience: Platform on-call engineers responding to a stale or failing NVD advisory feed.

Doc reconciliation (2026-05-30): This runbook was reconciled against the implementation in src/Concelier/__Libraries/StellaOps.Concelier.Connector.Nvd. Several commands and config keys in the original draft did not exist in code and have been corrected. The NVD connector is API-key-free — it pulls the public NIST CVE 2.0 REST feed via a sliding modified-window with no authentication. Sections that describe API-key/rate-limit handling have been removed or marked NOT IMPLEMENTED.

Metadata

How the connector works (ground truth)

Understanding the pipeline makes diagnosis deterministic:

• The connector runs as three separate jobs wired through the Concelier job scheduler (NvdDependencyInjectionRoutine, JobRegistrationExtensions):
  • source:nvd:fetch — pulls a modified-window of CVE records (cron 6 */4 * * *, ~every 4h).
  • source:nvd:parse — validates the raw JSON against the NVD schema and stores DTOs (cron 16 */4 * * *).
  • source:nvd:map — maps DTOs into canonical advisories (cron 26 */4 * * *).
• No API key / no authentication. The HTTP client (NvdServiceCollectionExtensions) targets https://services.nvd.nist.gov/rest/json/cves/2.0, sets a 30s timeout, User-Agent: StellaOps.Concelier.Nvd/1.0, Accept: application/json, and restricts AllowedHosts to the endpoint host. The configuration contributor (NvdConfigurationContributor) is explicit: vendor-specific shape: no auth.
• Fetch is a sliding modified-window. Each fetch advances a time window (lastModStartDate/lastModEndDate) sized by windowSize with windowOverlap, paginating with resultsPerPage=2000 + startIndex. On first run / empty state it looks back initialBackfill.
• Failure handling. On a fetch exception the connector records a failure via the source-state repository with a 5-minute retry backoff and re-throws. Schema-validation failures quarantine the offending document (status Failed) rather than failing the whole window, unless the payload is recoverable (has a vulnerabilities array).

Symptoms

• [ ] NVD feed sync failing or stale (freshness status stale/unavailable on the advisory-source freshness API)
• [ ] One of the source:nvd:fetch|parse|map job runs reporting Failed
• [ ] Errors in Concelier logs: NVD fetch failed for {Uri}, NVD schema validation failed, or Failed to parse NVD JSON payload
• [ ] Vulnerability data missing or outdated
• [ ] Connector metrics show rising nvd.fetch.failures / nvd.parse.failures / nvd.parse.quarantine

The original draft referenced an alert ConnectorNvdSyncFailed and a metric connector_sync_failures_total{source="nvd"}. Neither exists in source. The connector emits counters under the meter StellaOps.Concelier.Connector.Nvd (see Metrics). Wire alerts off those counters or off the freshness API.

Impact

Diagnosis

All HTTP examples target the Concelier WebService. Read endpoints (/jobs*, /api/v1/advisory-sources*) and the trigger endpoint require a bearer token. See Authorization for the required scopes.

Quick checks

1. Check source freshness (last sync, last success, last error):

   curl -s -H "Authorization: Bearer $TOKEN" \
     https://<concelier-host>/api/v1/advisory-sources/nvd/freshness | jq
   

   Look for: lastSyncAt, lastSuccessAt, lastError, errorCount, freshnessStatus (healthy / warning / stale / unavailable).

2. Check the aggregate source dashboard counters:

   curl -s -H "Authorization: Bearer $TOKEN" \
     https://<concelier-host>/api/v1/advisory-sources/summary | jq
   

3. Check the NVD job definitions and recent runs:

   curl -s -H "Authorization: Bearer $TOKEN" \
     https://<concelier-host>/jobs/definitions/source:nvd:fetch | jq
   curl -s -H "Authorization: Bearer $TOKEN" \
     "https://<concelier-host>/jobs/definitions/source:nvd:fetch/runs?limit=10" | jq
   

   Repeat for source:nvd:parse and source:nvd:map. Look for Failed outcomes and error messages.

Deep diagnosis

1. Inspect Concelier service logs for the NVD source. Grep for the structured log messages emitted by the connector:

   • NVD fetch failed for {Uri} — transport/HTTP failure (followed by a 5-minute backoff).
   • NVD window {Start} - {End} returned 304 — upstream unchanged (not an error).
   • NVD schema validation failed ... quarantined — payload failed schema (nvd.parse.quarantine).
   • Failed to parse NVD JSON payload — malformed JSON (nvd.parse.failures).

2. Check connector metrics (meter StellaOps.Concelier.Connector.Nvd, exported via OTLP/Prometheus — see Metrics):

   • nvd.fetch.failures rising → fetch/transport problem.
   • nvd.fetch.unchanged high → upstream returning 304 (usually benign).
   • nvd.parse.quarantine / nvd.parse.failures rising → schema/parse problem.

3. Check for an NVD outage / endpoint problem. Confirm the base endpoint is reachable from the Concelier host:

   curl -s -o /dev/null -w '%{http_code}\n' \
     'https://services.nvd.nist.gov/rest/json/cves/2.0?resultsPerPage=1'
   

   Also check: https://nvd.nist.gov/general/news

4. Confirm the active endpoint and pacing. The runtime overlay (NvdRuntimeOverlay) merges DB-persisted baseEndpoint, windowSize, windowOverlap, and initialBackfill over the configured values, so an operator override may have changed the effective endpoint. Inspect the advisory-source record (Console Admin → Advisory Sources, or the /api/v1/advisory-sources list) to see the active configuration.

Resolution

Immediate mitigation

1. Force a fetch/parse/map cycle (e.g. after a transient outage clears). Trigger the three jobs in order:

   curl -s -X POST -H "Authorization: Bearer $TOKEN" \
     -H 'Content-Type: application/json' -d '{"trigger":"runbook"}' \
     https://<concelier-host>/jobs/source:nvd:fetch
   # then, after fetch completes:
   curl -s -X POST -H "Authorization: Bearer $TOKEN" \
     -H 'Content-Type: application/json' -d '{"trigger":"runbook"}' \
     https://<concelier-host>/jobs/source:nvd:parse
   curl -s -X POST -H "Authorization: Bearer $TOKEN" \
     -H 'Content-Type: application/json' -d '{"trigger":"runbook"}' \
     https://<concelier-host>/jobs/source:nvd:map
   

   Responses: 202 Accepted (queued, Location: /jobs/{runId}), 404 (job not registered), 423 Locked (job disabled), 409 Conflict (a run is already active).

2. If you see persistent 304s with stale data, the cursor is at the head of the window — this is normal when there are genuinely no new modifications. Widen recovery by temporarily increasing initialBackfill only if the cursor was reset / data was lost.

3. Air-gap / disaster recovery: load from an offline feed snapshot. The supported offline path is the deterministic feed-snapshot bundle, not a per-source loader:

   stella feeds snapshot import <bundle-file> --validate
   

   (See stella feedsfor the full snapshot command set.)

Root cause fix

If the upstream endpoint changed or you need a private NVD mirror:

Override the base endpoint via the advisory-source runtime configuration (persisted setting key baseEndpoint, surfaced in Console Admin → Advisory Sources). The runtime overlay re-projects it onto NvdOptions on the next resolution; no redeploy is required. Configuration file equivalent (concelier:sources:nvd):

concelier:
  sources:
    nvd:
      baseEndpoint: "https://<your-nvd-mirror>/rest/json/cves/2.0"
      windowSize: "PT4H"      # 4h sliding modified-window
      windowOverlap: "PT5M"   # must be < windowSize
      initialBackfill: "P7D"  # look-back on first run / empty state

NOT IMPLEMENTED: auth / api-key, maxDocumentsPerFetch, fetchTimeout, requestDelay, sync_interval, delta_sync, and proxy config keys referenced in older docs/drafts. NvdOptions exposes only baseEndpoint, windowSize, windowOverlap, and initialBackfill. The HTTP timeout (30s) is fixed in NvdServiceCollectionExtensions. There is no NVD API-key authentication.

If hammering the upstream causes throttling:

Increase windowSize (fewer, larger windows) and/or rely on the built-in 5-minute failure backoff. The connector does not perform per-request rate-limit accounting; pacing is governed entirely by the job cron schedule and the modified-window size.

If network/firewall is the issue:

1. Verify outbound HTTPS connectivity to the configured baseEndpoint host from the Concelier host (the HTTP client restricts AllowedHosts to that host, so a redirected/mirrored host must match the configured endpoint).
2. Egress proxy configuration is a deployment/runtime concern (container env / HTTPS_PROXY), not an NVD connector config key.

If data parsing failures / quarantine spikes:

1. The connector validates against an embedded NVD CVE 2.0 JSON schema (NvdSchemaProvider). A spike in nvd.parse.quarantine usually means an upstream schema change. Quarantined documents are marked Failed; the window cursor still advances for recoverable payloads.
2. If NVD changed its schema, the connector must be updated in code (NvdSchemaProvider / NvdMapper) and redeployed. There is no stella connector ... schema-check or stella upgrade --component command.

Verification

# Re-trigger the pipeline (see Immediate mitigation) and poll the run:
RUN=$(curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"trigger":"runbook"}' \
  https://<concelier-host>/jobs/source:nvd:fetch -D - | awk -F'/jobs/' '/Location/{print $2}' | tr -d '\r')
curl -s -H "Authorization: Bearer $TOKEN" \
  https://<concelier-host>/jobs/$RUN | jq '.status'

# Confirm freshness recovered (lastSuccessAt advanced, errorCount stable):
curl -s -H "Authorization: Bearer $TOKEN" \
  https://<concelier-host>/api/v1/advisory-sources/nvd/freshness | jq

# Confirm no NVD failure log lines in the last hour (service log search).

Prevention

• [ ] Monitoring: Alert on freshnessStatus = stale/unavailable from /api/v1/advisory-sources/nvd/freshness, and on rising nvd.fetch.failures / nvd.parse.failures.
• [ ] Pacing: Keep windowOverlap < windowSize and large enough to cover upstream publish delays (default 5m overlap on a 4h window).
• [ ] Redundancy: Run overlapping connectors (GHSA, OSV) for cross-coverage.
• [ ] Offline: Maintain periodic feed snapshots (stella feeds snapshot create) for air-gap / disaster recovery.

Authorization

Scope constants live in src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs (ConcelierJobsRead = "concelier.jobs.read", ConcelierJobsTrigger = "concelier.jobs.trigger"). The advisory-source read endpoints are tenant-scoped (RequireTenant()).

Metrics

Meter: StellaOps.Concelier.Connector.Nvd (registered in Concelier WebService telemetry). All counters are long:

CLI surface

The primary NVD-relevant CLI is the deterministic feed-snapshot group (FeedsCommandGroup, wired as stella feeds snapshot ...):

stella feeds snapshot create [--label <l>] [--sources nvd ...] [--json]
stella feeds snapshot list [--limit <n>] [--json]
stella feeds snapshot export <snapshot-id> --output <file> [--compression zstd|gzip|none]
stella feeds snapshot import <input-file> [--validate]
stella feeds snapshot validate <snapshot-id>

Feed administration is also available via the admin feeds group (AdminCommandGroup.BuildFeedsCommand, backed by AdminCommandHandlers):

stella admin feeds list [--format table|json]
stella admin feeds status [--source <id>]          # e.g. --source nvd
stella admin feeds refresh [--source <id>] [--force]
stella admin feeds history --source <id> [--limit <n>]

Note — stella config feeds list|status is a stub. A second feed group exists under config (CommandFactory.BuildConfigFeedsCommand), but it prints hardcoded sample data (fixed timestamps, fabricated advisory counts) — do not use it for live diagnosis. Use stella admin feeds status or the freshness API instead.

NOT IMPLEMENTED (verified against src/Cli):

• stella connector test|credentials|logs|nvd ... and stella connector config set — there is no top-level connector command group at all.
• stella upgrade --component — there is no upgrade command group.
• stella vuln query — the vuln group does exist, but its subcommands are observations, list, show, assign, comment, accept-risk, verify-fix, target-fix, reopen, simulate, export; there is no query subcommand.
• stella offline load — the offline group does exist, but the kit loader is stella offline import (with status); there is no load subcommand.

stella doctor [run] --check <id> exists, but there is no NVD connector check id to pass it (no NVD check is registered under src/Doctor).

Related Resources

• Connector code: src/Concelier/__Libraries/StellaOps.Concelier.Connector.Nvd/
• Job registration / cron: src/Concelier/StellaOps.Concelier.WebService/Extensions/JobRegistrationExtensions.cs
• Architecture: docs/modules/concelier/connectors.md
• Connector ops doc: docs/modules/concelier/operations/connectors/nvd.md (⚠ stale — still documents an auth/api-key section and maxDocumentsPerFetch/fetchTimeout/requestDelay keys that do not exist in NvdOptions; reconcile separately)
• Related runbooks: connector-ghsa.md, connector-osv.md
• Dashboard: Grafana > Stella Ops > Feed Connectors