Test Framework Consolidation Plan

Audience: test-automation and platform engineers consolidating the Stella Ops test estate. Purpose: turn the current test inventory into a consistent, predictable, manifest-driven framework — one source of truth for which suites run, where, and when.

The companion documents are TEST_MANIFEST.md (the implemented manifest contract), TEST_CI_ROUTING.md (trigger placement), and TEST_CI_EVIDENCE_DASHBOARD.md (the CI evidence standard).

Snapshot date: 2026-05-22 UTC

Root-tree consolidation (2026-05-30): the operator overrode the earlier “keep production/operator harnesses at the repo root” stance. As of 2026-05-30 there are no test code, fixture, or corpus directories at the repo root. Everything cross-cutting now lives under src/__Tests/ (e.g. the E2E analyzer corpus at src/__Tests/e2e/scanner-analyzer-corpus/, operator Playwright at src/__Tests/playwright/, reachability/fixtures/supply-chain under src/__Tests/, and shared seed data under src/__Tests/__Datasets/); module-owned suites live under src/<Module>/__Tests/; and former root samples/ moved to docs/samples/. Tables and prose below that still describe tests/** at the root are historical baseline; the current standard is the src/__Tests/ layout. See sprint SPRINT_20260530_008_Tests_root_data_consolidation.md.

This plan turns the current test inventory into a consistent, predictable, and durable test framework. It is based on:

• docs/technical/testing/TEST_PROJECT_CI_INVENTORY.md
• docs/technical/testing/TEST_SUITE_CONSOLIDATION_DEEP_DIVE.md
• docs/technical/testing/test-project-ci-inventory.csv
• docs/technical/testing/test-suite-layer-inventory.csv
• docs/technical/testing/test-fixture-corpus-inventory.csv
• docs/technical/testing/test-e2e-scenario-inventory.csv

Current baseline:

• 666 project/runner/corpus rows.
• 597 runnable suite rows.
• 198 fixture/corpus roots.
• 364 browser E2E and Scanner SCM scenario rows.
• 368 high-value suites are not currently blocking CI.
• 23 runnable rows have no active CI mapping.
• 147 .NET suites have no static Category trait.
• 10 suites contain 34 static Skip = declarations.

Target State

Testing must become manifest-driven, not filename/glob-driven.

The target system has:

1. A versioned test manifest that is the source of truth for runnable suites, seed/fixture roots, CI placement, ownership, runtime cost, and gating state.
2. A change-impact planner that maps each commit to the smallest correct test plan and prints why each suite was selected or skipped.
3. Common project and seed-data structure across modules.
4. One canonical trait taxonomy enforced by tooling.
5. CI lanes separated by trigger type: per commit/PR, merge to main, nightly, weekly/release, and manual/live.
6. Explicit handling for expensive, live, hardware, Docker-heavy, and demo/sample test surfaces so they are visible but do not over-trigger.
7. Stable result artifacts and evidence metadata for every CI run.

Non-Goals

• Do not make every test PR-blocking.
• Do not make docs/demo/sample changes trigger product-wide test suites.
• Do not rely on *.Tests.csproj naming alone for discovery.
• Do not hide manual/live tests because they are expensive.
• Do not treat static inventory counts as behavioral verification.

Operating Decisions

• Primary .NET filter key: Category. run-test-category.sh already filters on Category=<value>, and StellaOps.TestKit.TestCategories already defines the main taxonomy.
• Lane becomes a CI scheduling concept, not the primary test filter key. Existing Lane traits are compatibility input during migration.
• Every suite must have a manifest row before it can be relied on by CI.
• Every fixture/corpus root must have an owner, refresh policy, and offline determinism policy.
• Every CI workflow must declare whether failures are blocking, report-only, or manual-only.
• The planner output is an artifact. CI must publish the selected suites and the skip reasons for non-selected suites.

Common Tooling

Create these durable tools and artifacts:

The scripts can be shell, PowerShell, or Python, but the interface must be stable and documented. CI should invoke the same entry points developers use locally.

Manifest Model

Each suite row should include:

suites:
  - id: scanner.reachability.unit
    owner: Scanner
    path: src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/StellaOps.Scanner.Reachability.Tests.csproj
    runner: dotnet
    category: Integration
    layers: [integration, golden, determinism]
    scenarioFamilies: [scanner/reachability/vex]
    affectedBy:
      paths:
        - src/Scanner/**
        - src/__Libraries/StellaOps.Reachability*/**
        - src/__Tests/e2e/scanner-analyzer-corpus/**
      fixtures:
        - scanner-analyzer-corpus
    runtime:
      tier: pr-targeted
      cost: medium
      infra: [postgres]
      timeoutMinutes: 15
    ci:
      pr: targeted-blocking
      main: targeted-blocking
      nightly: full-blocking
      manual: allowed
    evidence:
      resultFormat: trx
      artifacts:
        - TestResults/**

Each fixture/corpus row should include:

fixtures:
  - id: scanner-analyzer-corpus
    owner: Scanner
    path: src/__Tests/e2e/scanner-analyzer-corpus/fixtures
    type: scm-corpus
    offline: true
    refreshCommand: devops/scripts/e2e/provision-corpus.ps1 -Seed 42
    consumers:
      - scanner.e2e.analyzer-coverage
      - scanner.reachability.unit
    updatePolicy: explicit-review
    expectedHashPolicy: manifest-recorded

Common Project Structure

Standardize project placement as follows:

Rules:

• Every runnable test project must end in .Tests.csproj, .IntegrationTests.csproj, .E2ETests.csproj, .Benchmarks.csproj, or be explicitly registered as an exception in the manifest.
• Test support libraries must not look like runnable test projects.
• There is no top-level tests/** tree. Cross-product, production/operator, external-harness, and generated-corpus verification all live under src/__Tests/** (consolidated 2026-05-30); module-owned suites live under src/<Module>/__Tests/**.
• New test projects must reference or justify not using StellaOps.TestKit.
• New suites require a manifest row in the same change.

Common Seed and Fixture Structure

Standardize fixture roots by intent:

Every non-trivial fixture root needs a local fixtures.manifest.yml:

id: concelier-nvd-fixtures
owner: Concelier
type: connector-fixture
offline: true
source: frozen-upstream-sample
refreshCommand: src/Concelier/devops/refresh-nvd-fixtures.ps1
updatePolicy: explicit-review
consumers:
  - concelier.connector.nvd.tests
expectedHashPolicy: manifest-recorded

Rules:

• Fixture refresh must be explicit, never accidental during normal tests.
• Live/upstream refresh tests are Live and never PR-blocking.
• Golden updates require a named env var or command, plus review of the diff.
• Generated corpuses must have a seed, generator version, and ignored output policy when they are too large to commit.
• License or provenance notes are required when fixture data is copied from upstream sources.

Trait and Category Taxonomy

Every .NET test class or method must have exactly one primary Category unless the manifest records a temporary exception.

Canonical primary categories:

Secondary traits:

Skip rules:

• No indefinite Skip = without a sprint/task reference and reason.
• Live tests should use opt-in guards, not static skips.
• Quarantined flaky tests need expiry and owner.

CI Placement

Per Commit / Pull Request

Purpose: fast, correct feedback for the changed surface.

Always run:

• Manifest/schema validation.
• Change-impact planner.
• Test project discovery drift check.
• Affected build graph compile.
• Unit/contract/security/static tests selected by the planner.
• Fixture manifest validation when fixtures changed.

Conditionally run:

• Targeted integration tests when code, migrations, persistence, API, or Testcontainers-backed suites are affected.
• Web Vitest changed-spec drift for Web changes.
• Small Playwright smoke only when Web route/navigation/critical UX paths or Playwright specs changed.
• Docs sample/template tests only when those samples/templates changed.
• Runner self-tests when .gitea/scripts/test/** or test workflow files change.

Do not run automatically on every commit:

• Full generic .NET sweep.
• Full production/operator Playwright.
• Full Scanner generated SCM corpus.
• Long benchmarks.
• Live upstream connector tests.
• Hardware or OS-confinement tests.

Merge To Main

Purpose: catch integration misses after merge without making every PR pay the full cost.

Run after push to main:

• The same planner against the merged diff.
• Expanded targeted matrix for affected modules.
• Changed fixture consumers.
• Architecture and OpenAPI conformance.
• Security and determinism suites touched by the merged change.
• Any PR report-only suite that was selected by the planner.

Main-merge failures should block release promotion and produce an owner-routed failure artifact. They should not silently disappear into report-only logs.

Nightly

Purpose: complete regression and expensive deterministic coverage.

Run:

• Full manifest sweep, sharded by runtime cost and infra class.
• Cross-cutting src/__Tests/Integration/** and src/__Tests/e2e/**.
• Scanner SCM analyzer coverage against the deterministic local Gitea target.
• Production/operator Playwright against the controlled live stack.
• Air-gap, offline, chaos, determinism, golden, and interop suites.
• Fixture drift and manifest drift checks.
• Performance smoke with thresholds.

Nightly should publish full coverage evidence and open/refresh failure tracking, but only suites marked nightly: blocking should block release readiness.

Weekly / Release

Purpose: long, costly, or environment-dependent validation.

Run:

• BenchmarkDotNet suites.
• Long load/performance suites.
• External live connector smoke.
• OS confinement and hardware-backed crypto/provider tests.
• Compatibility matrices that are too expensive for nightly.

Manual

Purpose: controlled investigation, live target validation, and ad hoc release evidence.

Manual workflows must still consume the manifest so their scope, owners, and artifacts are visible.

Change-Impact Planner

The planner replaces broad implicit triggering with an auditable decision.

Inputs:

• Changed files.
• Test manifest.
• Fixture manifest.
• Dependency map from project references and package references.
• Optional prior failure state.

Outputs:

• test-plan.json
• test-plan.md
• selected suites with trigger reason
• skipped suites with skip reason
• required runners and infra
• estimated runtime and shard plan
• gating mode per suite

Examples:

The planner must avoid the current binary affects_all behavior where every shared change either fans out too much or defers too much to nightly.

CI Enablement Priorities

1. Register all 23 ci-gap rows in the manifest.
2. Split Web into separate rows: Vitest, source-served Playwright, docker/live Playwright, live-search Playwright, and Air-gap notification UI specs.
3. Add production/operator Playwright under src/__Tests/playwright as nightly/manual first, with PR smoke only for changes to src/__Tests/playwright/** or shared UI routes it exercises.
4. Add explicit rows for src/__Tests/e2e/** and src/__Tests/Integration/**.
5. Keep former top-level tests/Telemetry rows under owning module test roots; the P0 Telemetry metrics coverage now lives in the module-owned Telemetry core test project, and the Doctor plugin storage project has been moved under src/Doctor/__Tests.
6. Add BinaryIndex benchmark rows as weekly/manual, not PR.
7. Decide whether docs sample/template tests are real runnable examples or excluded sample artifacts. Until then, trigger them only on sample/template path changes.
8. Convert current report-only .NET PR fanout to targeted non-blocking, then to blocking by layer once trait coverage and planner accuracy are proven.

Phased Rollout

Phase 1 - Manifest Foundation

Deliver:

• TEST_MANIFEST.yml
• schema validation
• fixture manifest schema
• manifest linter
• generated inventory comparison

Exit criteria:

• All 597 runnable suite rows are represented or explicitly excluded.
• All 198 fixture/corpus roots are represented or explicitly excluded.
• CI fails if new runnable suites or fixture roots appear without manifest updates.

Phase 2 - Change Planner

Deliver:

• plan-test-run tool.
• Markdown and JSON plan artifacts.
• Planner dry-run job for PRs.
• Rules for docs, samples, fixtures, module code, shared libraries, workflows, and global build files.

Exit criteria:

• A docs-only change selects no product suites.
• A docs sample change selects only sample/template checks.
• A module change selects only manifest-related module suites and required shared dependencies.
• Shared-library changes produce a representative PR slice and a full nightly follow-up plan.

Phase 3 - Trait and Structure Migration

Deliver:

• Reconciled docs for Category, Lane, and TestKit traits.
• Category coverage for the 147 trait-missing .NET rows.
• Skip audit closure for the 10 suites with static skips.
• Project structure exceptions recorded in the manifest.

Exit criteria:

• run-test-category.sh can use trait filters for migrated suites.
• No new test project can merge without Category/owner metadata.
• Static skips require owner, reason, sprint/task reference, and expiry.

Phase 4 - CI Enablement

Deliver:

• PR targeted workflow consuming test-plan.json.
• Main-merge workflow consuming test-plan.json.
• Nightly full manifest sweep, sharded by runtime/infra.
• Manual/live workflows consuming manifest rows.
• Result normalization and artifact naming.

Exit criteria:

• The 23 ci-gap rows are no longer invisible.
• High-value suites move from report-only to explicit blocking, release-blocking, nightly-blocking, or manual-only.
• No workflow relies on implicit glob discovery as the primary source of truth.

Phase 5 - Fixture and Corpus Governance

Deliver:

• fixtures.manifest.yml files for major fixture roots.
• Refresh commands for Scanner SCM corpus, reachability fixtures, Concelier connectors, docs/QA fixtures, and major golden corpuses.
• Hash or snapshot policy for deterministic fixtures.
• Live refresh lanes separated from offline deterministic regression lanes.

Exit criteria:

• Fixture changes route to consuming suites only.
• Corpus refresh changes are reviewable and reproducible.
• Large generated corpuses remain deterministic without being accidentally committed or accidentally skipped.

Phase 6 - Gate Hardening

Deliver:

• Blocking policy per suite and per trigger.
• Runtime budgets and shard definitions.
• Failure owner routing.
• Quarantine process with expiry.
• Release-readiness summary from PR, main, nightly, and release lanes.

Exit criteria:

• Release readiness cannot claim green status while selected blocking suites are failing or skipped.
• Report-only suites are visible and intentionally classified.
• CI runtime stays bounded because the planner avoids unrelated full-suite reruns.

Consistency Rules

• New test suite without manifest row: fail.
• New fixture root without fixture manifest: fail.
• New [Fact] / [Theory] without Category: fail after migration window.
• Live test in PR plan: fail unless explicitly manual-only.
• New static Skip = without task reference and expiry: fail.
• Test project under docs/** or sample/template path cannot trigger product suites unless manifest explicitly declares a product dependency.
• CI workflow that runs tests without publishing test-plan and result summary: fail once migration is complete.

Open Decisions

• Whether production/operator Playwright should become nightly-blocking or release-blocking after stabilization.
• Which high-value family moves to blocking first: cross-cutting integration/E2E, Scanner reachability/VEX, Concelier connector fixtures, or security/identity.
• Whether docs sample/template projects should stay runnable or be converted to static compile-only examples.
• Whether Lane traits should be mechanically removed after Category coverage is complete, or retained as aliases for older docs.

Recommended First Sprint Order

1. Manifest foundation and linter.
2. Change planner dry-run.
3. CI-gap registration.
4. Web runner split.
5. Trait migration for the largest/highest-value suites.
6. Fixture manifest rollout for Scanner, Concelier, and cross-cutting __Tests.
7. Gate hardening from report-only to blocking by proven lane.

Implementation State

The first implementation slice created the manifest foundation:

• TEST_MANIFEST.yml seeded from the suite, fixture, and E2E inventories.
• TEST_MANIFEST.schema.json and FIXTURE_MANIFEST.schema.json.
• refresh-test-manifest.py, validate-test-manifest.py, plan-test-run.py, and run-test-plan.py.
• .gitea/workflows/test-manifest-plan.yml for manifest/planner CI validation once pushed.

Current local evidence is recorded in TEST_MANIFEST.md and docs/implplan/SPRINT_20260522_008_DOCS_testing_framework_consolidation.md.