Test Suite Consolidation Deep Dive

Audience: maintainers planning the test-consolidation effort — deciding which suites become PR-blocking, nightly, release-gating, or remain report-only.

Snapshot date: 2026-05-22 UTC.

This report extends TEST_PROJECT_CI_INVENTORY.mdwith a consolidation-grade static inventory of runnable suites and fixture/corpus ownership. It is intended to support consolidation planning — it reports structure and CI mapping, not behavioral correctness.

Generated appendices:

• test-suite-layer-inventory.csv
• test-fixture-corpus-inventory.csv
• Existing scenario appendix: test-e2e-scenario-inventory.csv

Follow-up consolidation plan: TEST_FRAMEWORK_CONSOLIDATION_PLAN.md.

Method

The inventory was generated from repository-local files only. It excludes bin, obj, node_modules, dist, generated artifacts, Playwright reports, coverage output, and test result directories.

For .NET projects, estimated_test_methods_or_spec_files is a static count of [Fact] plus [Theory] methods, not expanded data rows. Data-driven breadth is captured separately through InlineData, MemberData, and ClassData counts in the CSV. For frontend, Playwright, and Go rows, the same field counts spec/test files.

Headline Findings

CI State

The three automatic rows are the OpenAPI conformance pack, the Air-gap Federation integration pack, and root docs/API parity tooling. Frontend is intentionally conservative in this table because its current single row mixes Vitest PR drift, default Playwright, and Air-gap E2E mappings; the manifest work should split those into separate runner rows.

Suite Layers

Layer counts are non-exclusive because a single suite can be both integration and security, or golden and determinism.

Scenario Families

The highest fragmentation is in advisory-feed connector fixtures and Scanner reachability/VEX coverage. These should be manifest-owned first because they combine fixture corpuses, external-format contracts, and high-value regression coverage.

Consolidation Buckets

CI Gap Rows

Largest Suites

These suites should not be moved casually. They are the best candidates for a manifest-driven split into fast deterministic slices, fixture-backed golden slices, and slower infra/live slices.

Fixture and Corpus Ownership

Top fixture roots by file count:

Fixture root counts by owner are concentrated in Concelier (57), Scanner (38), cross-cutting __Tests (17), Policy (9), docs/qa (9), Attestor (8), and Authority (7). Consolidation should add ownership, refresh command, offline status, and determinism policy for each root.

Skip Audit

The deep inventory found 10 suites with static Skip = attributes, totaling 34 skipped declarations. The affected modules include __Tests, AdvisoryAI, Cli, Concelier, Platform, ReleaseOrchestrator, Router, and Scanner. These should be reviewed before using suite green status as release evidence.

Consolidation Recommendations

1. Create a versioned test manifest and make CI consume it. Each row should name the runner, layer, owner, fixture roots, infra, trigger, and gating state.
2. Resolve ci-gap rows first. Product/test-surface priorities are production Playwright, src/__Tests/e2e/**, src/__Tests/Integration/**, module-owned Doctor plugin tests, and BinaryIndex benchmarks.
3. Split mixed rows before changing gates. The Web row currently combines Vitest specs, default Playwright, live/search Playwright, and Air-gap federation UI specs. Those should become separate manifest entries.
4. Normalize Category / Lane traits before relying on filter-based consolidation. There are 147 .NET rows with no static Category trait.
5. Promote high-value report-only suites in stages: first make them explicit non-blocking lanes with stable artifacts, then decide which slices become PR blocking, nightly blocking, release blocking, or permanently report-only.
6. Give fixture corpuses first-class ownership. The Scanner SCM corpus, reachability fixtures, Concelier connector fixtures, and docs/QA fixtures should have refresh commands, offline rules, and expected deterministic outputs.
7. Keep expensive, live, hardware, OS-confinement, and Docker-heavy tests out of generic PR fanout unless they have fast slices. They still need manifest rows so consolidation does not hide them.

Decisions Needed Later

• Whether production/operator Playwright under src/__Tests/playwright should become PR-blocking, scheduled, release-blocking, or remain manual.
• Whether docs sample/template test projects should be runnable CI examples or excluded from the consolidation manifest.
• Which high-value report-only layer should be promoted first: cross-cutting integration/e2e, Scanner reachability/VEX, Concelier connector fixtures, or security/identity/crypto.