Runbook: Feed Connector - OSV (Open Source Vulnerabilities) Failures

Audience: Platform on-call engineers diagnosing and recovering the OSV advisory feed connector in Concelier (the vulnerability-feed ingestion service of the Stella Ops release control plane). Use this runbook when OSV ingestion is failing, stale, or producing incomplete open-source ecosystem coverage.

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-006 - Feed Connector Runbooks

Doc reconciliation (2026-05-30): This runbook was reconciled against the implementation in src/Concelier/__Libraries/StellaOps.Concelier.Connector.Osv, the CLI in src/Cli/StellaOps.Cli, the Doctor checks in src/Doctor, and the scope catalog StellaOpsScopes.cs. Several commands, config keys, metrics, and the Doctor check in the original draft did not exist in code and have been corrected or marked NOT IMPLEMENTED. Key correction: the OSV connector is anonymous, public-bucket HTTP — it downloads per-ecosystem all.zip archives over plain HTTPS GET from osv-vulnerabilities.storage.googleapis.com. There is no GCS authentication, no service-account credentials, no proxy config key, and no separate “OSV API” vs “GCS bulk” path — it is one path.

Metadata

FieldValue
ComponentConcelier / OSV Connector (StellaOps.Concelier.Connector.Osv, source id osv)
SeverityHigh
On-call scopePlatform team
Last updated2026-05-30
Doctor checkNOT IMPLEMENTED — no check.connector.osv-health Doctor check exists (verified against src/Doctor; there is no Concelier/connector Doctor plugin at all). Use stella admin feeds status --source osv, the job-run endpoints, and the connector metric below instead.

How the connector works (ground truth)

Understanding the pipeline makes diagnosis deterministic:

Symptoms

The original draft referenced an alert ConnectorOsvSyncFailed and a metric connector_sync_failures_total{source="osv"}. Neither exists in source. The only OSV-specific metric is osv.map.canonical_metric_fallbacks under the meter StellaOps.Concelier.Connector.Osv. Wire alerts off that counter or off the feeds-status / job-run state.

Impact

Impact TypeDescription
User-facingOpen source ecosystem vulnerabilities may be missed
Data integrityData becomes stale; no data loss (failures back off 10 min and retry; the per-ecosystem cursor only advances on processed records)
SLA impactVulnerability currency SLO violated for affected ecosystems

Diagnosis

Job-trigger and job-run endpoints target the Concelier WebService and require a bearer token. See Authorization for the required scopes.

Quick checks

  1. Check OSV feed sync status (CLI):

    stella admin feeds status --source osv

    This calls GET /api/v1/admin/feeds/osv/status on the backend. (If the command returns a transport/404 error, the server-side admin-feeds endpoint may not be wired in your build; fall back to the job-run checks below, which exercise the connector directly.)

  2. Check the OSV job definitions and recent runs:

    curl -s -H "Authorization: Bearer $TOKEN" \
  https://<concelier-host>/jobs/definitions/source:osv:fetch | jq
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<concelier-host>/jobs/definitions/source:osv:fetch/runs?limit=10" | jq

    Repeat for source:osv:parse and source:osv:map. Look for Failed outcomes and error messages.

  3. Test OSV bucket connectivity from the Concelier host:

    curl -s -o /dev/null -w '%{http_code}\n' \
  'https://osv-vulnerabilities.storage.googleapis.com/PyPI/all.zip'

    Also check: https://osv.dev/ for an upstream outage.

NOT IMPLEMENTED: stella doctor --check check.connector.osv-health, stella connector test osv, and stella connector osv api-status. There is no connector CLI command group and no OSV Doctor check. stella doctor [run] --check <id> exists, but there is no OSV connector check id to pass it. To list the real checks: stella doctor list.

Deep diagnosis

  1. Inspect Concelier service logs for the OSV source. Grep for the structured log messages emitted by the connector:

    • OSV fetch failed for ecosystem {Ecosystem} — transport/HTTP failure for that ecosystem’s archive (followed by a 10-minute failure backoff).
    • Failed to parse OSV entry {Entry} for ecosystem {Ecosystem} — a single record in the archive failed to deserialize (skipped, not fatal).
    • Failed to deserialize OSV document {DocumentId} / OSV document {DocumentId} produced empty payload — parse-stage quarantine (document marked Failed).
    • OSV map checkpoint paused after {N} documents — map hit its time budget and will resume next run (informational, not an error).

    The collected/diagnostic log view is available via stella admin diagnostics logs --service <svc> --level error --tail 200.

  2. Check the OSV connector metric (meter StellaOps.Concelier.Connector.Osv, exported via OTLP/Prometheus — see Metrics):

    • osv.map.canonical_metric_fallbacks rising → OSV is publishing severity-only / CVSS v4-only advisories that hit the canonical-metric fallback path. Usually benign; alert only on large spikes (see the connector ops doc thresholds).

  3. Confirm the active base URI and ecosystem list. The connector binds concelier:sources:osv to OsvOptions. If an operator overrode BaseUri (e.g. to a private mirror) or trimmed Ecosystems, a “missing ecosystem” symptom is configuration, not an outage. Inspect the effective configuration (Console Admin → Advisory Sources, or the configuration file under concelier:sources:osv).

Resolution

Immediate mitigation

  1. Force a fetch/parse/map cycle (e.g. after a transient outage clears). The supported, code-backed trigger is the Concelier db fetch CLI, which queues the source:osv:<stage> jobs:

    stella db fetch --source osv --stage fetch
# then, after fetch completes:
stella db fetch --source osv --stage parse
stella db fetch --source osv --stage map

    Equivalent raw job triggers:

    curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"trigger":"runbook"}' \
  https://<concelier-host>/jobs/source:osv:fetch

    Responses: 202 Accepted (queued, Location: /jobs/{runId}), 404 (job not registered), 423 Locked (job disabled), 409 Conflict (a run already active).

    The stella admin feeds refresh --source osv [--force] CLI also exists and posts to the backend admin-feeds refresh endpoint; prefer db fetch when you need stage-level control.

    NOT IMPLEMENTED: stella admin feeds refresh --source osv --ecosystem npm. The refresh command accepts only --source and --force — there is no --ecosystem option. Fetch always iterates the full configured ecosystem list (capped by MaxAdvisoriesPerFetch). Also NOT IMPLEMENTED: stella connector osv sync-from-gcs (there is no separate GCS sync — the normal fetch is the GCS download).

  2. If you see persistent 304 Not Modified with stale data, the cached ETag / Last-Modified is at the head of the archive — normal when there are genuinely no new modifications. Only widen recovery (increase InitialBackfill) if the cursor was reset / data was lost.

  3. Air-gap / disaster recovery: load from an offline kit. The supported offline path is the offline-kit import, not a per-source loader. The kit is passed via --bundle (the bundle is a .tar.zst payload); the manifest defaults to manifest.json next to the bundle:

    stella offline import --bundle <kit-bundle.tar.zst> [--manifest <manifest.json>] [--dry-run]

    By default import verifies the DSSE signature and Rekor receipt (--verify-dsse/--verify-rekor, both on by default); --dry-run validates the kit without activating it. Check current kit state with stella offline status.

    NOT IMPLEMENTED: stella offline load --source osv --package <file>, and stella offline import <kit-file> as a positional argument. There is no offline load command and no --source/--package options; the command is stella offline import and it takes the kit via the required --bundle (-b) option, not a positional path. The only offline subcommands are import and status.

Root cause fix

If the bucket / endpoint is unreachable or you need a private mirror:

Override the base URI via the advisory-source configuration (concelier:sources:osv); the connector re-binds OsvOptions on resolution. Example configuration:

concelier:
  sources:
    osv:
      baseUri: "https://<your-osv-mirror>/"   # must be an absolute URI
      archiveFileName: "all.zip"               # per-ecosystem archive name
      ecosystems: ["PyPI", "npm", "Maven", "Go", "crates.io",
                   "Hex", "Packagist", "RubyGems", "Chainguard", "Wolfi"]
      maxAdvisoriesPerFetch: 250
      initialBackfill: "14.00:00:00"           # look-back on first/empty run
      modifiedTolerance: "00:10:00"
      requestDelay: "00:00:00.250"
      httpTimeout: "00:03:00"
      mapRunTimeBudget: "00:15:00"

NOT IMPLEMENTED: osv.proxy, osv.gcs_auth, osv.gcs_credentials, osv.ecosystems.disabled, and sync_interval config keys referenced in older drafts. OsvOptions exposes only BaseUri, Ecosystems, InitialBackfill, ModifiedTolerance, MaxAdvisoriesPerFetch, ArchiveFileName, RequestDelay, HttpTimeout, and MapRunTimeBudget. The bucket is anonymous/public; there is no GCS authentication. Egress proxy is a deployment concern (container env / HTTPS_PROXY), not an OSV connector config key.

If a specific ecosystem is failing:

Temporarily remove the offending ecosystem from concelier:sources:osv:ecosystems (the list must keep at least one entry, or OsvOptions.Validate() throws). There is no stella connector config set osv.ecosystems.disabled command and no stella connector osv ecosystem-check command — edit the configuration and re-resolve.

If parsing / deserialization failures spike:

A spike in parse-stage Failed documents usually means an upstream OSV schema change. The connector tolerates per-record parse failures (skips them); a systemic change requires updating the DTO / mapper (OsvVulnerabilityDto, OsvMapper) in code and redeploying. There is no stella connector osv schema-check and no stella upgrade --component connector-osv command.

Verification

# Re-trigger the pipeline (see Immediate mitigation):
stella db fetch --source osv --stage fetch
stella db fetch --source osv --stage parse
stella db fetch --source osv --stage map

# Poll the most recent run state:
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<concelier-host>/jobs/definitions/source:osv:map/runs?limit=1" | jq '.[0].status'

# Confirm feed status recovered:
stella admin feeds status --source osv

# Spot-check a recent OSV advisory via the vuln observations API:
stella vuln observations --tenant <tenant> --alias OSV-2026-xxxx --json

# Confirm no OSV failure log lines in the last hour (service log search).

NOT IMPLEMENTED: stella vuln query OSV-2026-xxxx and stella admin feeds status --source osv --watch. The vuln command exposes observations (filter by --alias/--purl/etc.), not query, and feeds status has no --watch flag (--watch exists on stella doctor).

Prevention

Authorization

OperationEndpointRequired scope (claim)
Trigger OSV jobs (stella db fetch)POST /jobs/source:osv:{fetch,parse,map}concelier.jobs.trigger (policy Concelier.Jobs.Trigger)
List jobs / definitions / runsGET /jobs, GET /jobs/definitions/{kind}, GET /jobs/definitions/{kind}/runs, GET /jobs/{runId}, GET /jobs/activeconcelier.jobs.read or concelier.jobs.trigger (policy Concelier.Jobs.Read; trigger implies read)
Run canonical merge (stella db merge)POST /jobs/merge:reconcileconcelier.jobs.trigger (policy Concelier.Jobs.Trigger) — not concelier.merge (see note)
Read advisory observations (stella vuln observations)GET /concelier/observations, GET /advisories/observationsvuln:view (policy Concelier.Observations.Read), tenant-scoped

Corrected (2026-05-30): Earlier drafts claimed stella db merge required the concelier.merge scope and that observations needed advisory:read at GET /api/v1/vuln/observations*. Verified against src/Concelier/StellaOps.Concelier.WebService/Program.cs:

  • Merge is dispatched through the same catch-all job-trigger endpoint (POST /jobs/{*jobKind} with jobKind=merge:reconcile), which is guarded by the Concelier.Jobs.Trigger policy — i.e. the required scope is concelier.jobs.trigger, the same as every other job trigger. The StellaOpsScopes.ConcelierMerge constant ("concelier.merge") exists in the catalog but is not wired to any Concelier endpoint — it does not gate the merge job.
  • Observations are served by Concelier at GET /concelier/observations and GET /advisories/observations (the CLI stella vuln observations calls the former; see ConcelierObservationsClient.BuildRequestUri). Both are guarded by the Concelier.Observations.Read policy, which is registered with scope StellaOpsScopes.VulnView ("vuln:view") — not advisory:read. There is no /api/v1/vuln/observations* route on Concelier.

Scope constants live in src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs (ConcelierJobsRead = "concelier.jobs.read", ConcelierJobsTrigger = "concelier.jobs.trigger", VulnView = "vuln:view", AdvisoryRead = "advisory:read"; ConcelierMerge = "concelier.merge" exists but is unused by Concelier endpoints).

Metrics

Meter: StellaOps.Concelier.Connector.Osv (version 1.0.0), exported via OTLP/Prometheus.

MetricTypeMeaning
osv.map.canonical_metric_fallbackscounter (advisories)Incremented at map time when a mapped advisory has empty CVSS metrics but still has a non-empty canonical metric id (OsvMapper.BuildSeverityCanonicalMetricId). The id has the form <resolved-source>:severity/<severity>, where <resolved-source> is the per-ecosystem alias the record was attributed to — e.g. npm:severity/HIGH, pypi:severity/..., rustsec:severity/..., or osv:severity/... when the ecosystem is unmapped (not always literally osv:). Tags: canonical_metric_id, severity (falls back to unknown when the advisory has no severity), ecosystem, reason=no_cvss.

This is the only OSV-connector-specific metric. The connector_sync_failures_total{source="osv"} metric in the original draft does not exist. For failure signal use the source:osv:* job-run outcomes and feed status.

See docs/modules/concelier/operations/connectors/osv.md for canonical-metric fallback dashboard/alert guidance (baseline volume is currently <5/day; alert when the 1-hour sum exceeds 50 for any ecosystem).

CLI surface

CommandPurposeStatus
stella db fetch --source osv --stage {fetch,parse,map} [--mode <m>]Trigger a connector stage (queues source:osv:<stage>)Implemented
stella db mergeRun canonical merge reconciliationImplemented
stella admin feeds status --source osvShow OSV feed sync status (GET /api/v1/admin/feeds/osv/status)CLI implemented (server endpoint may be a gap in some builds)
stella admin feeds refresh --source osv [--force]Trigger an OSV feed refreshCLI implemented
stella admin feeds list | history --source osvList feeds / show sync historyCLI implemented
stella sources check osv [--json]Check connectivity to an advisory sourceImplemented
stella sources status / list / enable osv / disable osvSource configuration managementImplemented
stella vuln observations --tenant <t> --alias OSV-...Inspect raw advisory observations (calls GET /concelier/observations, scope vuln:view)Implemented
stella offline import --bundle <kit-bundle.tar.zst> [--manifest <m>] [--dry-run]Import an offline kit (required --bundle; no positional path)Implemented
stella offline statusShow current offline kit statusImplemented

NOT IMPLEMENTED (invented in the earlier draft): stella connector test osv, stella connector osv ecosystems status, stella connector osv api-status, stella connector osv gcs-status, stella connector osv sync-from-gcs, stella connector osv api-test, stella connector osv gcs-test, stella connector osv ecosystem-check, stella connector osv schema-check, stella connector logs osv, stella connector config set osv.*, stella offline load --source osv, stella vuln query, and stella upgrade --component connector-osv. There is no connector command group, no upgrade command, and no per-source offline loader.