Concelier CCCS Connector Operations

Last updated: 2026-05-31

Audience: Stella Ops operators running the Concelier CCCS connector.

This runbook covers day-to-day operation of the Canadian Centre for Cyber Security connector (runtime/catalog source ID cccs, fetch pipeline source:cccs:*), including configuration, telemetry, and historical backfill guidance for the English and French advisory feeds.

1. Configuration Checklist

Network egress (or mirrored cache) for https://www.cyber.gc.ca/ and the JSON API endpoints under /api/cccs/.
Set the Concelier options before restarting workers. Example concelier.yaml snippet:

concelier:
  sources:
    cccs:
      feeds:
        - language: "en"
          uri: "https://www.cyber.gc.ca/api/cccs/threats/v1/get?lang=en&content_type=cccs_threat"
        - language: "fr"
          uri: "https://www.cyber.gc.ca/api/cccs/threats/v1/get?lang=fr&content_type=cccs_threat"
      maxEntriesPerFetch: 80        # increase temporarily for backfill runs
      maxKnownEntries: 512
      requestTimeout: "00:00:30"
      requestDelay: "00:00:00.250"
      failureBackoff: "00:05:00"

Note. The /api/cccs/threats/v1/get endpoint returns thousands of records per language (≈5,100 rows each as of 2025-10-14). The connector honours maxEntriesPerFetch, so leave it low for steady-state and raise it for planned backfills.

Feed binding behaviour (Sprint 20260512_016). CccsOptions.Feeds is a getter-only IList<>. IConfiguration.Bind() appends new feed entries to the existing list rather than replacing it. If you supply only Concelier__Sources__Cccs__Feeds__0__Uri (without __Language), the new entry inherits the property-initializer default Language="en" and would previously have collided with another en entry. The connector now (a) seeds canonical en/fr defaults lazily – only when no feeds are supplied – and (b) re-derives the language from the URI’s lang= query parameter when present, then deduplicates by language (last-write-wins). The hard-fail Duplicate CCCS feed language configured exception has been removed.

2. Telemetry & Logging

Metrics (Meter StellaOps.Concelier.Connector.Cccs):
- cccs.fetch.attempts, cccs.fetch.success, cccs.fetch.failures
- cccs.fetch.documents, cccs.fetch.unchanged
- cccs.parse.success, cccs.parse.failures, cccs.parse.quarantine
- cccs.map.success, cccs.map.failures
Shared HTTP metrics via SourceDiagnostics:
- concelier.source.http.requests{concelier.source="cccs"}
- concelier.source.http.failures{concelier.source="cccs"}
- concelier.source.http.duration{concelier.source="cccs"}
Structured logs
- CCCS fetch completed feeds=… items=… newDocuments=… pendingDocuments=…
- CCCS parse completed parsed=… failures=…
- CCCS map completed mapped=… failures=…
- Warnings fire when document payloads/DTOs go missing or parser sanitisation fails.

Suggested Grafana alerts:

increase(cccs.fetch.failures_total[15m]) > 0
rate(cccs.map.success_total[1h]) == 0 while other connectors are active
histogram_quantile(0.95, rate(concelier_source_http_duration_bucket{concelier_source="cccs"}[1h])) > 5s

3. Historical Backfill Plan

Snapshot the source – the API accepts page=<n> and lang=<en|fr> query parameters. page=0 returns the full dataset (observed earliest date_created: 2018-06-08 for EN, 2018-06-08 for FR). Mirror those responses into Offline Kit storage when operating air-gapped.
Stage ingestion:
- Temporarily raise maxEntriesPerFetch (e.g. 500) and restart Concelier workers.
- Run chained jobs until pendingDocuments drains: stella db fetch --source cccs --stage fetch, then --stage parse, then --stage map.
- Monitor cccs.fetch.unchanged growth; once it approaches dataset size the backfill is complete.
Optional pagination sweep – for incremental mirrors, iterate page=<n> (0…N) while response.Count == 50, persisting JSON to disk. Store alongside metadata (language, page, SHA256) so repeated runs detect drift.
Language split – keep EN/FR payloads separate to preserve canonical language fields. The connector emits Language directly from the feed entry, so mixed ingestion simply produces parallel advisories keyed by the same serial number.
Throttle planning – schedule backfills during maintenance windows; the API tolerates burst downloads but respect the 250 ms request delay or raise it if mirrored traffic is not available.

4. Selector & Sanitiser Notes

CccsHtmlParser now parses the unsanitised DOM (via AngleSharp) and only sanitises when persisting ContentHtml.
Product extraction walks headings (Affected Products, Produits touchés, Mesures recommandées) and consumes nested lists within div/section/article containers.
HtmlContentSanitizer allows <h1>…<h6> and <section> so stored HTML keeps headings for UI rendering and downstream summarisation.

5. Fixture Maintenance