Design Proposal: MITRE ATT&CK / D3FEND Ingest Scope Re-Evaluation

DECIDED: Path 2 — implementations removed in Sprint 20260513_008. The PM chose Path 2 (remove) over this proposal’s recommended Path 1 (status quo + safety regression test). All ATT&CK / D3FEND ingest code, tests, plugin manifests, source definitions, HTTP clients, job anchors, and seed rows have been deleted. Forward migrations 028 + 029 reverse the live-DB effects of migrations 017 and 027 (mirror domain). See docs/implplan/SPRINT_20260513_008_Concelier_mitre_path2_removal.md for the removal evidence trail.

The analysis below is the original proposal, preserved for the record. It recommends Path 1; that recommendation was superseded by the decision in the banner. Read it for the options reasoning, not for current state.

Status: Decided — Path 2 (remove). See banner above. Date: 2026-05-12 (drafted) / 2026-05-13 (decision + execution) Sprint: docs/implplan/SPRINT_20260505_039_Concelier_mitre_connector_implementation.md — CON-MITRE-004 (closed by Path 2) docs/implplan/SPRINT_20260513_008_Concelier_mitre_path2_removal.md — execution sprint Authors: Concelier Product / Platform Architecture

Context

Sprint 20260505_046 shipped two Concelier connectors — Connector.MitreAttack (STIX 2.1 attack-pattern bundle) and Connector.MitreD3fend (SPARQL ontology mappings) — that ingest MITRE ATT&CK technique IDs (e.g. T1055.011) and D3FEND defensive technique keys (e.g. D3F-TokenBinding) into vuln.advisories. Both connectors are wired into DI, seeded in 017_seed_mitre_connector_sources.sql, and have green adjacent test projects (6/6 each).

On 2026-05-12 the PM reopened Sprint 039 because three concerns survived the original “ship it” decision:

  1. Zero downstream consumers exist. No policy rule, risk-engine knob, findings query, or VEX flow reads mitre-attack / mitre-d3fend source IDs. The morning rationale (“keep ingest warm for future consumers”) is speculative.
  2. The “advisory” shape is being repurposed. Both parsers emit Advisory records with affectedPackages = Array.Empty<>(), no CVSS, no severity. Distinction is carried in vuln.sources.config as {"category":"ttp"} / {"category":"defensive"} — a string flag, not a schema boundary.
  3. Sprint 20260512_006 introduced a CVE-aware policy gate (PolicyRuntimeEvaluator + CveDenylistExtractor). ATT&CK rows now share a table with the data the gate reads from. If the gate matched loosely, an operator denylist rule could accidentally trip on a technique ID.

This proposal walks Paths 1 / 2 / 3 against the actual code paths and recommends a course.

Current state

What Sprint 046 shipped:

What persists from the connectors:

Who reads vuln.advisories:

Who consumes category=ttp / category=defensive: nobody outside the two connectors themselves. Verified by Grep across src/ and docs/.

Options analysis

AspectPath 1 (status quo)Path 2 (remove)Path 3 (extract to threat module)
Code churnnone (+ 1 regression test)~17 files deleted + sln + migration + docsnew StellaOps.Concelier.Threat library, new threat.techniques schema, copy parser/fetcher logic, update plugin registry
Schema integrityT-IDs share vuln.advisories but the safety boundary is the absent advisory_affected rowsclean — vuln.advisories is CVE-only by constructionclean — threat.techniques is a new namespace
CVE-gate accidental match risknone today (no advisory_affected rows, gate joins through that table); future contributors must preserve “no affected rows” invariantnone (ATT&CK absent)none (different table)
Loses ingested datanoyes (techniques disappear; STIX bundle hash + raw payload preserved in vuln.documents until that table is cleared)no (data moves)
Future consumer onboardingtrivial — just JOIN vuln.advisories ON source_id = 'mitre-attack' (with the “no affected_rows” caveat)requires a focused 1-week sprint to re-add connectors with a real consumer’s shape in mindtrivial in the threat namespace; cross-namespace queries are explicit
Architectural cleanlinessmediocre — “advisory” abstraction is overloadedhigh — Concelier is “vulnerability advisories only”high — Concelier remains vuln-only; threat-intel has its own home
Reversibilityhigh — Path 2 / Path 3 still available laterlow — re-implementation cost is non-trivial when a real consumer surfacesmedium — schema migration cost grows with row count
Effort estimate~0.5 day (test only)~1 day (deletion + sln cleanup + docs)~5–8 days (new library + schema + reorg + tests + migration of seeded rows)

Path 1 (status quo): retain + add safety-boundary regression test

Leave the connectors in place. Add one regression test asserting:

  1. MitreAttackParser.ToAdvisory(...) produces an Advisory with AffectedPackages.Count == 0.
  2. Same for MitreD3fendParser.ToAdvisory(...).
  3. (Optional, broader) An integration test that loads an ATT&CK technique into vuln.advisories, runs a finding.cve_id == "T1055" denylist rule through PolicyRuntimeEvaluator, and asserts the decision is allow because no advisory_affected row exists.

Pros: zero deletion risk; data stays warm; tests lock the safety invariant. Cons: the architectural smell (“technique IDs in vuln.advisories”) persists. Future contributors who add an affected_packages block to the parsers — even with good intent — break the invariant.

Path 2 (remove)

Delete Connector.MitreAttack and Connector.MitreD3fend libraries, delete the seed migration (or add 019_drop_mitre_connector_sources.sql that revokes the rows), remove the SourceDefinitions entries, update docs.

Pros: cleanest schema; no smell, no accidental-match story. Cons: throws away working implementation + raw payload archive. Re-implementation later requires re-licensing/re-vetting the upstream JSON shape, re-writing parser + tests, re-seeding sources. Empirical observation: the morning rationale (“warm ingest”) is reversible — but only at non-trivial cost.

Path 3 (extract to threat module)

Create StellaOps.Concelier.Threat (or StellaOps.Findings.Threat) with:

Pros: architecturally clean; future threat-aware policy rules have a well-named home; Concelier’s contract becomes “vulnerability advisories only” again. Cons: largest effort. Builds a parallel persistence story that has no consumer today, just to clean an internal smell. Speculative infrastructure.

Recommendation

Path 1 (status quo) + add the safety-boundary regression test.

Rationale (short form):

Migration plan if Path 1 is chosen

No source-code changes required to the connectors. Add:

  1. Regression test in src/Concelier/__Tests/StellaOps.Concelier.Connector.MitreAttack.Tests/:
    • MitreAttackParserTests.ToAdvisory_DoesNotEmitAffectedPackages — asserts Advisory.AffectedPackages.Count == 0 for a representative technique.
    • Mirror test for D3FEND in StellaOps.Concelier.Connector.MitreD3fend.Tests/.
  2. Integration regression test (optional, in src/Policy/__Tests/StellaOps.Policy.Engine.Tests/):
    • Seed a vuln.advisories row with primary_vuln_id = 'T1055', source_id = mitre-attack source UUID, no advisory_affected row.
    • Compile a policy bundle with rule deny when finding.cve_id == "T1055".
    • Resolve a release whose components have ATT&CK as a known source.
    • Assert decision == "allow" and matches array is empty.
  3. Doc update: add a “Safety boundary” subsection to docs/modules/concelier/connectors/credentials-matrix.md (MITRE section) noting:
    • Why T-IDs in vuln.advisories cannot leak into policy denylists.
    • The invariant: ATT&CK / D3FEND parsers MUST NOT emit affected_packages rows.
  4. AGENTS.md addition in the two connector libraries pointing to the invariant doc — so future contributors don’t add affected-package inference helpfully.

Open questions for PM

  1. Should the seeded sources be enabled = false until a real consumer ships? Currently both run on the standard fetch cadence. Flipping enabled = false in the seed migration reduces the live row count and network footprint, with no behaviour change for the absent consumer.
  2. Should the UI hide category=ttp / category=defensive sources from the operator catalog until a consumer surfaces? Today they’re visible in /api/v1/advisory-sources, which is confusing — an operator can “disable” them but nothing downstream notices.
  3. Where does the Path 3 trigger live? Suggest: file a docs-only ADR (ADR-MITRE-THREAT-EXTRACTION-TRIGGER.md) that explicitly says “when the first non-Concelier consumer of mitre-attack / mitre-d3fend is filed, immediately schedule the Path 3 extraction sprint.” That makes the deferral auditable.
  4. What’s the policy gate’s posture on aliased denylists? Today CveDenylistExtractor matches on finding.cve_id only — would a future finding.alias_contains "T1055" rule shape change the safety story? (Not today, but worth flagging.)

Decision log