No-Merge Migration Playbook

Last updated: 2025-11-06 (reconciled against src/Concelier 2026-05-30)

Purpose. Retire the legacy advisory Merge service (AdvisoryMergeService) and cut every advisory pipeline over to Link-Not-Merge (LNM) — observations plus linksets — with zero data loss and a rehearsed rollback.

Audience. The BE-Merge, Architecture, DevOps, and Docs guilds coordinating Sprint 110 (Ingestion & Evidence). This playbook feeds CONCELIER-LNM-21-101 / MERGE-LNM-21-001 and downstream DOCS-LNM-22-008.

Reconciliation note (2026-05-30): the master toggle (concelier:features:noMergeEnabled, default true), the gated AddMergeModule DI registration, the merge:reconcile job removal, the allowlist behaviour, the CONCELIER0002 analyzer, and the LNM observation/linkset read APIs are all live in src/Concelier and verified against source. Several operator-facing surfaces in this playbook are not yet implemented and are flagged inline as NOT IMPLEMENTED (e.g. the stella concelier CLI verbs, MergeJobScheduler.PauseAsync, the policy:overlays:requireLinksetEvidence flag) or corrected to the real source names (metric instruments, allowlist values, persistence engine). Do not script against the NOT IMPLEMENTED items until they ship.

0. Scope & objectives

• Primary goal: cut over all advisory pipelines to Link-Not-Merge with no residual dependencies on AdvisoryMergeService.
• Secondary goals: maintain deterministic evidence, zero data loss, and reversible deployment across online and offline tenants.
• Success criteria:
  • All connectors emit observation affected.versions[] with provenance and pass LNM guardrails.
  • Linkset coverage dashboards show no advisories missing normalized version ranges. (Note: the metric names missing_version_entries_total / version_entries_total and a Normalized version rules missing… log warning are NOT IMPLEMENTED as named instruments. The CERT/CC connector emits a per-source histogram certcc.map.normalized_versions.count describing “normalized version rules emitted per CERT/CC advisory”; the LNM cache emits lnm.cache.hit_total, lnm.cache.write_total, and lnm.cache.rebuild_ms from StellaOps.Concelier.WebService/Telemetry/LinksetCacheTelemetry.cs. Treat the names below as placeholders pending an observability spike.)
  • Policy, Export Center, and CLI consumers operate solely on observations/linksets. The current LNM read surface is GET /v1/lnm/linksets, POST /v1/lnm/linksets/search, and GET /v1/lnm/linksets/{advisoryId} (registered in Program.cs; tenant authorization enforced in-handler via EnsureTenantAuthorized). GET /concelier/observations (auth policy Concelier.Observations.Read, scope vuln:view) still exists but is itself DEPRECATED — DeprecationMiddleware/DeprecationHeaders.LegacyConcelierObservations mark it with successor /v1/lnm/linksets?includeObservations=true. New consumers must target the /v1/lnm/linksets* endpoints; the legacy merged-document path is gated off by default.
  • Rollback playbook validated and rehearsed in staging.

1. Prerequisites checklist

Do not proceed to Phase 1 until all prerequisites are checked or explicitly waived by Architecture guild.

2. Feature flag & configuration plan

2025-11-06: WebService now defaults concelier:features:noMergeEnabled to true, skipping Merge DI registration and removing the merge:reconcile job unless operators set the flag to false and allowlist the job (MERGE-LNM-21-002).

2025-11-06: Analyzer CONCELIER0002 ships with Concelier hosts to block new references to AdvisoryMergeService / AddMergeModule. Suppressions must be paired with an explicit migration note. 2025-11-06: Analyzer coverage validated via unit tests catching object creation, field declarations, typeof, and DI extension invocations; merge assemblies remain exempt for legacy cleanup helpers.

Configuration hygiene: Document the toggle values per environment in ops/devops/configuration/staging.md and ops/devops/configuration/production.md. Air-gapped customers receive defaults through the Offline Kit release notes.

3. Rollout phases

3.1 Environment sequencing

1. Dev/Test clusters: Validate all automation. Run full regression suite (dotnet test src/Concelier/...).
2. Staging: Execute complete backfill (see §4) and collect 24 h of telemetry before sign-off.
3. Production: Perform cutover during low-ingest window; communicate via Slack/email + status page two days in advance.
4. Offline kit: Package new Observer snapshots with LNM-only data; ensure instructions cover flag toggles for air-gapped deployments.

3.2 Smoke test matrix

• Confirm the deployment reports healthy and shows zero Merge workers. (stella concelier status --include linkset is NOT IMPLEMENTED — there is no concelier CLI group; check the WebService health endpoint and confirm the scheduler has no merge:reconcile definition, which is the default behaviour when noMergeEnabled=true.)
• Run a policy evaluation against sample tenants and confirm identical findings pre/post cutover. (The exact verb stella policy evaluate is NOT IMPLEMENTED, and there is no policy check or policy list verb either — those do not exist in CommandFactory.cs. The policy CLI group’s actual verbs include simulate, lint, compile, activate, edit, test, new, history, explain, init, version, submit, review, publish, rollback, sign, verify-signature, lattice, verdicts, and promote. For a parity-style evaluation use stella policy simulate <policy-id> or the Policy WebService evaluate endpoint.)
• Export Center bundle diff shows only expected metadata changes (manifest ID, timestamps).
• Grafana dashboards: LNM cache instruments (lnm.cache.write_total, lnm.cache.rebuild_ms) steady and concelier.merge.identity_conflicts flatlined after cutover. (The earlier names linkset_insert_duration_ms / merge.identity.conflicts are NOT the emitted instrument names — see §5.)

4. Backfill strategy

1. Freeze Merge writes: Prevent new merge events while snapshots are taken. (MergeJobScheduler.PauseAsync is NOT IMPLEMENTED — no such type/method exists. With noMergeEnabled=true (the default) the merge:reconcile job is never scheduled at all; to freeze a deployment that still has Merge enabled, set noMergeEnabled=true and restart, or remove merge:reconcile from the allowlist.)
2. Generate linkset baseline: Rebuild linksets from the advisory_raw store (table verified in StellaOps.Concelier.Persistence). (The CLI/host command ... -- linkset backfill --from <date> is NOT IMPLEMENTED — there is no linkset backfill verb on the WebService host or CLI; the only orchestrator backfill is the generic stella orch backfill start <source-id> --from --to, which is a per-source ingestion backfill, not a linkset rebuild. Treat the linkset-rebuild job as roadmap and capture whatever artefacts the chosen mechanism produces.) Capture job output artefacts and attach to the sprint issue.
3. Validate parity: Compare sample advisories’ linkset vs merged output. (The internal diff tool tools/concelier/compare-linkset-merge.ps1 is NOT IMPLEMENTED — no tools/concelier/ directory or that script exists in the repo. Until a parity tool ships, drive the comparison from the read APIs in §0 plus a manual/scripted diff.) Any diffs must be triaged before production cutover.
4. Publish evidence: For air-gapped tenants, create a one-off Offline Kit slice and push to staging mirror. (No linkset-backfill export profile exists in source — export profile linkset-backfill is NOT IMPLEMENTED as a named profile; pick a concrete Export Center profile or capture the artefacts manually.)
5. Tag snapshot: Record the PostgreSQL snapshot/WAL position (Concelier persists to PostgreSQL via StellaOps.Concelier.Persistence, not MongoDB — the original “Mongo oplog” reference is incorrect) and the S3/object-storage manifests (federation bundle export uses S3/RustFS via S3FederationBundlePublisher) in ops/devops/runbooks/concelier/no-merge.md (new section) so rollback knows the safe point.

Determinism: rerunning the backfill with identical inputs must produce byte-identical linkset documents. Use the --verify-determinism flag where available and archive the checksum report under artifacts/lnm-backfill/<date>/.

5. Validation gates

• Metrics (actual instrument names — verified in source): lnm.cache.hit_total, lnm.cache.write_total, lnm.cache.rebuild_ms (meter StellaOps.Concelier.Linksets, from LinksetCacheTelemetry.cs) and concelier.merge.identity_conflicts (meter StellaOps.Concelier.Merge, from AdvisoryMergeService.cs). The names linkset_insert_duration_ms, linkset_documents_total, and normalized_version_rules_missing are NOT IMPLEMENTED as emitted instruments; a “normalized rules emitted” signal exists only as a per-source histogram in the CERT/CC connector — certcc.map.normalized_versions.count (unit rules, “Distribution of normalized version rules emitted per CERT/CC advisory”, CertCcDiagnostics.cs).
  • Gate (revised): confirm linkset cache writes (lnm.cache.write_total) are flowing and concelier.merge.identity_conflicts stays flat before relying on noMergeEnabled; the original normalized_version_rules_missing == 0 gate is NOT IMPLEMENTED (substitute the CERT/CC histogram or an explicit coverage check). Original window: 48 h before enabling noMergeEnabled.
• Logs: Ensure no merge-fallback log lines after cutover. (The literal string Fallbacking to merge service does NOT exist in source; instead confirm the startup warning “Legacy merge module disabled via concelier:features:noMergeEnabled; Link-Not-Merge mode active.” is present, per Program.cs, and that merge:reconcile job logs are absent.)
• Change streams: Policy and Scheduler should observe only advisory.linkset.updated@1 events (contract name verified in AdvisoryLinksetUpdatedEvent.cs); monitor for stragglers referencing merge IDs.
• QA: The StellaOps.Concelier.Merge.Tests project exists; the off-by-default behaviour is asserted by WebServiceEndpointsTests (MergeModuleDisabledByDefault, MergeModuleReenabledWhenFeatureFlagCleared, MergeJobRemovedWhenAllowlistExcludes, MergeJobRemainsWhenAllowlisted) and the CONCELIER0002 analyzer is covered by StellaOps.Concelier.Merge.Analyzers.Tests. Add integration tests verifying LNM-only exports.

Capture validation evidence in the sprint journal (attach Grafana screenshots + CLI output).

6. Rollback plan

1. Toggle sequence:
   • Set concelier:features:noMergeEnabled=false.
   • Re-enable the Merge job by allowlisting it: concelier:features:mergeJobAllowlist=["merge:reconcile"] (the legacy section concelier:jobs:merge:allowlist is also honoured). The job kind is merge:reconcile, not merge:default— merge:default does not exist anywhere in source and will not re-enable anything. (With noMergeEnabled=false and no allowlist, the job is also re-added automatically.)
   • The policy:overlays:requireLinksetEvidence step is moot: that flag is NOT IMPLEMENTED (no enforcement to disable).
2. Data considerations:
   • Linkset writes continue, so no data is lost; ensure Policy consumers ignore linkset-only fields during rollback window.
   • If Merge pipeline was fully removed (Phase 3 complete), redeploy the Merge service container image from the rollback tag published before cutover.
3. Verification:
   • Confirm Merge workers are active: check that the scheduler now contains the merge:reconcile job definition and the AdvisoryMergeService is DI-registered. (stella concelier status is NOT IMPLEMENTED — there is no concelier CLI group; use the WebService health/scheduler state instead.)
   • Monitor concelier.merge.identity_conflicts (not merge.identity.conflicts) for spikes; if present, roll forward and re-open incident with Architecture guild.
4. Communication:
   • Post incident note in #release-infra and customer status page.
   • Log rollback reason, window, and configs in ops/devops/incidents/<yyyy-mm-dd>-no-merge.md.

Rollback window should not exceed 4 hours; beyond that, plan to roll forward with a hotfix rather than reintroducing Merge.

7. Documentation & communications

• Update docs/modules/concelier/architecture.md appendix to mark Merge deprecated and link back to this playbook.
• Coordinate with Docs guild to publish operator-facing guidance under docs/releases/ (a dedicated release-notes-no-merge.md) and update CLI help text.
• Notify product/CS teams with a short FAQ covering timelines, customer impact, and steps for self-hosted installations.

8. Responsibilities matrix

9. Deliverables & artefacts

• Config diff per environment (stored in GitOps repo).
• Backfill checksum report (artifacts/lnm-backfill/<date>/checksums.json).
• Grafana export (PDF) showing validation metrics.
• QA test run attesting to LNM-only regressions passing.
• Updated runbook entry in ops/devops/runbooks/concelier/.

10. Migration readiness checklist

Update the checklist as each item completes; completion of every row is required before moving to Phase 2 (Cutover).

With this playbook completed, proceed to MERGE-LNM-21-002 to remove the Merge service code paths and enforce compile-time analyzers that block new merge dependencies.