Concelier OSV Connector - Operations Runbook

Last updated: 2025-10-16

This runbook is for operators who run the OSV connector in a Stella Ops Concelier instance. The OSV connector ingests advisories from OSV.dev across open-source ecosystems. This note highlights the merge/export expectations introduced with the canonical metric fallback behaviour, plus the dashboards, alerts, and fixtures that keep that behaviour observable.

1. Canonical metric fallbacks

When OSV omits CVSS vectors (common for CVSS v4-only payloads) the mapper now emits a deterministic canonical metric id in the form osv:severity/<level> and normalises the advisory severity to the same <level>.
Metric: osv.map.canonical_metric_fallbacks (counter) with tags severity, canonical_metric_id, ecosystem, reason=no_cvss. Watch this alongside merge parity dashboards to catch spikes where OSV publishes severity-only advisories.
Merge precedence still prefers GHSA over OSV; the shared severity-based canonical id keeps Merge/export parity deterministic even when only OSV supplies severity data.

2. CWE provenance

database_specific.cwe_ids now populates provenance decision reasons for every mapped weakness. Expect decisionReason="database_specific.cwe_ids" on OSV weakness provenance and confirm exporters preserve the value.
If OSV ever attaches database_specific.cwe_notes, the connector will surface the joined note string in decisionReason instead of the default marker.

3. Dashboards & alerts

Extend existing merge dashboards with the new counter:
- Overlay sum(osv.map.canonical_metric_fallbacks{ecosystem=~".+"}) with Merge severity overrides to confirm fallback advisories are reconciling cleanly.
- Alert when the 1-hour sum exceeds 50 for any ecosystem; baseline volume is currently <5 per day (mostly GHSA mirrors emitting CVSS v4 only).
Exporters already surface canonicalMetricId; no schema change is required, but ORAS/Trivy bundles should be spot-checked after deploying the connector update.

4. Runbook updates

Fixture parity suites (osv-ghsa.*) now assert the fallback id and provenance notes. Regenerate via dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj.
When investigating merge severity conflicts, include the fallback counter and confirm OSV advisories carry the expected osv:severity/<level> id before raising connector bugs.

5. Bulk backfill tuning (parse/map checkpointing)

OSV is a bulk, multi-ecosystem source. Pointing the connector at the real per-ecosystem all.zip feeds (rather than the conservative incremental window) queues on the order of 100k advisories per crawl, whose parse and map stages take well over an hour to drain. Two behaviours keep that crawl from being orphaned or timed out:

Job timeouts (WebService). source:osv:{fetch,parse,map} are registered with a 90-minute timeout and a 30-minute lease (JobRegistrationExtensions.cs), unlike the generic 15-minute/5-minute connector defaults. JobCoordinator cancels a run’s work via CancellationTokenSource.CancelAfter(definition.Timeout), so the smaller defaults cancelled a bulk parse/map mid-drain. The lease heartbeat fires every lease/2 (= 15 min), comfortably inside the 30-minute TTL, so a healthy long-running OSV run keeps its lease and is not superseded by the scheduler’s stale-run reconciler (which cancels rows whose lease has lapsed or that have been overtaken by a newer active run of the same kind).
Incremental checkpointing (connector). Both ParseAsync and MapAsync checkpoint progress to the cursor in batches and always flush on exit (normal completion, time-budget pause, or a propagating cancellation), so a cancelled stage keeps everything it had already completed instead of replaying — or orphaning — the whole queue. The parse stage moves documents from pendingDocuments to pendingMappings; the map stage drains pendingMappings.

Connector knobs (OsvOptions, all runtime-overridable via the OSV runtime overlay, defaults preserve single-pass behaviour for small incremental fetches):

Option	Default	Purpose
`ParseRunTimeBudget`	60 min	Max wall-clock a single parse run drains before checkpointing and deferring the remainder to the next run.
`ParseCheckpointBatchSize`	100	Parsed documents accumulated before a cursor checkpoint flush (turns per-run writes from O(n) into O(n/K)).
`ParseCheckpointInterval`	5 s	Forces a checkpoint flush after this wall-clock even if fewer than the batch size completed.
`MapRunTimeBudget`	15 min	Map-stage equivalent of `ParseRunTimeBudget`.
`MapCheckpointBatchSize`	100	Map-stage equivalent of `ParseCheckpointBatchSize`.
`MapCheckpointInterval`	5 s	Map-stage equivalent of `ParseCheckpointInterval`.

For a manual full crawl against the real feeds, the live override (devops/compose/docker-compose.osv-real.override.yml) points the connector at the real all.zip endpoints and disables the resume-window ratchet (ModifiedTolerance=36500.00:00:00, MaxAdvisoriesPerFetch=100000, RequestDelay=0) so the full corpus passes. Operationally, disabling the source’s scheduler during the crawl avoids any same-kind re-trigger contention; the raised timeouts + checkpointing make the crawl resumable even across restarts.