Concelier NKCKI Connector - Operations Runbook

Last updated: 2026-06-05

This runbook is for operators who run the NKCKI connector in a Stella Ops Concelier instance. It covers configuration, telemetry, archive backfill, and failure handling.

1. Overview

The NKCKI connector ingests JSON bulletin archives from the operator-approved NKCKI/NKTsKI mirror, expanding each *.json.zip attachment into per-vulnerability DTOs before canonical mapping. The default endpoint is the public safe-surf mirror (https://safe-surf.ru/specialists/bulletins-nkcki/) because it mirrors the NKCKI bulletin archives currently seeded in the source catalog; the canonical source remains disabled by default for export-control/operator approval.

Archive JSON roots may be either:

• a root array of vulnerability objects; or
• a wrapper object shaped like { "total": <count>, "data": [ ... vulnerability objects ... ] }.

Wrapper objects are fanned out through data[]. Root objects that do not carry a vulnerability identity (vuln_id.FSTEC or vuln_id.MITRE) are dropped and diagnosed rather than persisted as one synthetic advisory.

2. Configuration

Key options exposed through concelier:sources:ru-nkcki:http:

• maxBulletinsPerFetch – limits new bulletin downloads in a single run (default 5).
• maxListingPagesPerFetch – maximum listing pages visited during pagination (default 3).
• listingCacheDuration – minimum interval between listing fetches before falling back to cached artefacts (default 00:10:00).
• cacheDirectory – optional path for persisted bulletin archives used during offline or failure scenarios.
• requestDelay – delay inserted between bulletin downloads to respect upstream politeness.

When operating in offline-first mode, set cacheDirectory to a writable path (the default compose stack uses /var/lib/concelier/jobs/cache/ru-nkcki) and pre-populate bulletin archives via the offline kit. Operator-managed seed archives must retain mirror provenance in the source configuration/cache manifest; live fetches also stamp the configured mirror base into document metadata as ru-nkcki.mirror_base.

3. Telemetry

RuNkckiDiagnostics emits the following metrics under meter StellaOps.Concelier.Connector.Ru.Nkcki:

• nkcki.listing.fetch.attempts / nkcki.listing.fetch.success / nkcki.listing.fetch.failures
• nkcki.listing.pages.visited (histogram, pages)
• nkcki.listing.attachments.discovered / nkcki.listing.attachments.new
• nkcki.bulletin.fetch.success / nkcki.bulletin.fetch.cached / nkcki.bulletin.fetch.failures
• nkcki.entries.processed (histogram, entries)
• nkcki.archive.json_entries (histogram, entries)
• nkcki.records.parsed (counter, records)
• nkcki.records.dropped (counter, records, tagged by reason)
• nkcki.records.dropped.missing_key (counter, records)
• nkcki.records.duplicate_advisory_key_groups (histogram, groups)

Each processed archive also logs JSON entry count, wrapper-reported record count, data[] record count, parsed records, dropped records, missing-key drops, duplicate advisory-key groups, and newly enqueued documents. Integrate these counters into standard Concelier observability dashboards to track crawl coverage, cache hit rates, malformed-wrapper drops, and low-row symptoms.

4. Archive backfill strategy

Bitrix pagination surfaces archives via ?PAGEN_1=n. The connector now walks up to maxListingPagesPerFetch pages, deduplicating bulletin IDs and maintaining a rolling knownBulletins window. Backfill strategy:

1. Enumerate pages from newest to oldest, respecting maxListingPagesPerFetch and listingCacheDuration to avoid refetch storms.
2. Persist every *.json.zip attachment to the configured cache directory. This enables replay when listing access is temporarily blocked.
3. During archive replay, ProcessCachedBulletinsAsync enqueues missing documents while respecting maxVulnerabilitiesPerFetch. Safe-surf-style wrapper archives are expanded by data[], so the effective row count is the number of valid vulnerability objects in data[], not the wrapper object itself.
4. For historical HTML-only advisories, collect page URLs and metadata while offline. A full HTML and PDF extraction pipeline for these legacy advisories is future work and is not yet implemented.

For large migrations, seed caches with archived zip bundles, then run fetch/parse/map cycles in chronological order to maintain deterministic outputs.

5. Failure handling

• Listing failures mark the source state with exponential backoff while attempting cache replay.
• Bulletin fetches fall back to cached copies before surfacing an error.
• Cache path permission errors usually mean the source was pointed at an image-owned path such as /app/cache. In compose, use Concelier__Sources__Nkcki__CacheDirectory=/var/lib/concelier/jobs/cache/ru-nkcki.
• Malformed wrapper roots and rows missing vuln_id.FSTEC/vuln_id.MITRE are dropped with diagnostics instead of creating fallback Guid advisory keys.
• Integration tests use Testcontainers with PostgreSQL for connector verification.

Refer to ru-nkcki entries in src/Concelier/__Libraries/StellaOps.Concelier.Connector.Ru.Nkcki/TASKS.md for outstanding items.