Test Manifest

Audience: test-automation engineers and anyone wiring a suite into Stella Ops CI. Purpose: document the implemented test-manifest contract — its files, the required suite and fixture fields, the planner/validator/runner commands, and how the manifest drives CI placement.

The test manifest is the consolidation control point for suite ownership, fixture ownership, trigger routing, and CI gating. It is the source of truth that the change-impact planner and CI lanes consume; see TEST_FRAMEWORK_CONSOLIDATION_PLAN.md for the design rationale and TEST_CI_ROUTING.md for trigger placement.

Files

• TEST_MANIFEST.yml - generated manifest. It uses JSON syntax inside a .yml file so CI can parse it with the Python standard library only.
• TEST_MANIFEST.schema.json - suite and manifest field contract.
• FIXTURE_MANIFEST.schema.json - fixture entry contract for future split fixture manifests.
• .gitea/scripts/test/test_tree.py - the single definition of “exists in the tree” (git ls-files), “is a test project”, and the placement vocabulary; shared by the refresh and validation scripts.
• .gitea/scripts/test/refresh-test-inventory.py - reconciles the suite and fixture inventory CSVs with the tree: repoints a moved suite or fixture root, drops a withdrawn one, and adds a measured row for every test project that has none. --check writes nothing and exits 1 when an inventory is stale.
• .gitea/scripts/test/refresh-test-manifest.py - regenerates the manifest from the suite, fixture, and E2E inventory CSV snapshots. It keeps the existing manifest entry for every path still inventoried (ids, retirement notes and owner routing survive), carries a moved suite’s entry to its new path, generates only new rows, and then applies the placement rule below. --from-scratch ignores the existing manifest.
• .gitea/scripts/test/validate-test-manifest.py - validates manifest coverage, invalid trigger/gating combinations, tree currency and placement (see Tree currency and placement). --self-test proves each tree check still fires.
• .gitea/scripts/test/plan-test-run.py - creates test-plan.json and test-plan.md for a set of changed files.
• .gitea/scripts/test/run-test-plan.py - consumes test-plan.json and emits stable runner summaries. It is dry-run by default. --suite-timeout-minutes N runs each suite in its own process group and ends the group after N minutes: the suite is recorded as failed with exit code 124, and its log first lists the processes that were still running. test-manifest-execution.yml passes 45.
• .gitea/scripts/test/summarize-test-readiness.py - turns a runner summary into a release-readiness rollup and fails when selected blocking/required suites or validation checks failed.
• .gitea/scripts/test/render-test-dashboard.py - joins manifest, plan, run, and readiness artifacts into a Markdown/JSON/CSV execution matrix.
• .gitea/scripts/test/collect-gitea-test-history.py - uses GITEA_TOKEN from the environment to collect recent Gitea run/job history without storing credentials.
• .gitea/scripts/test/refresh-ci-gap-routing.py - regenerates the explicit CI-gap routing table from the manifest.
• .gitea/scripts/test/validate-ci-routing.py - validates CI-gap trigger placement and no-overtrigger constraints.
• .gitea/workflows/test-manifest-plan.yml - CI validation workflow for manifest and planner changes.
• .gitea/workflows/test-manifest-execution.yml - shared PR/main/nightly/weekly/manual entry point that consumes the planner and runner artifacts.

Required Suite Fields

Every runnable suite row must be represented or explicitly excluded.

Required Fixture Fields

Every fixture or corpus root must be represented or explicitly excluded.

Tree currency and placement

validate-test-manifest.py checks the manifest against the repository, not only against the inventory CSVs it was generated from. It fails when:

• a suite path, a suite fixtureRoots entry, an e2eScenarios path or a fixture path names nothing tracked in the tree;
• a tracked test project has no suite row. A test project is a tracked .csproj under src/, outside src/__Obsoleted/, whose name ends in .Tests or that sets <IsTestProject>true; <IsTestProject>false opts a .Tests name out. This is the convention src/Directory.Build.props applies;
• a suite runs in no automatic lane and records no reason. A suite runs when one of ci.pr, ci.main, ci.nightly or ci.weekly is not none or excluded: test-manifest-execution.yml runs pr and main in its targeted lane and nightly and weekly in its full sweep. Otherwise ci.noneReason must say why;
• a suite that runs still carries ci.noneReason, or a runs in dedicated lane: <workflow> reason names a workflow that has no push, pull_request or schedule trigger or does not name the project file.
• a suite runs in an automatic lane but its .csproj under src/ is not a test project: its name does not end in .Tests, it does not set <IsTestProject>true, and it references no Microsoft.NET.Test.Sdk (whose props set IsTestProject). dotnet test on such a project builds nothing, runs nothing and exits 0, so the lane reports a pass for a suite that never ran. A project named otherwise, such as *.IntegrationTests, sets IsTestProject, UseXunitV3 and TestingPlatformDotnetTestSupport itself, because src/Directory.Build.props is evaluated before the project body and derives them only from the .Tests suffix.

refresh-test-manifest.py places every suite that no inventory rule places:

1. An excluded suite records its retirement note or exclusion reason.
2. A suite that an event-triggered workflow other than the manifest lanes already runs records runs in dedicated lane: <workflow>. Such workflows supply infrastructure the sweep does not, such as Oracle or a live estate.
3. Every other suite runs in the nightly full sweep, report-only. The sweep sends suites whose runtime.infra names Testcontainers to the apparmor-host shard and the rest to the container shards.

Reviewed infrastructure metadata

An existing curated row survives a normal inventory/manifest refresh. Correcting a known fixture dependency therefore requires a reviewed remeasurement of that row: use refresh-test-inventory.py::measure_project for source counts/traits, then refresh-test-manifest.py::build_suite_manifest for the derived fields. Retain its suite ID, path, trigger policy and other curated decisions; run the normal refresh chain afterward. Do not regenerate the entire manifest from scratch to repair one suite.

PCD-370 applies this to scanner-stellaops-scanner-triage-tests. Its supported TriagePostgresFixture requires real PostgreSQL through Testcontainers, so the reviewed infrastructure is docker-testcontainers, postgres, testcontainers. The canonical cost rule classifies that fixture suite as medium; the existing host-infrastructure shard selects it and container shards exclude it. Nightly/manual remain report-only. The manifest validator rejects this exact active project’s missing infrastructure; its self-test includes absent/partial metadata and correct host/container routing in both modes. It does not infer infrastructure for other projects from their names.

The refreshed inventory counts 56 facts and one theory (57 declarations), including the fixture migration guard and Integration traits. These counts are not execution totals: the retained PCD-361 complete project run executed 67 tests with zero failures/skips. Planner selection proves placement; it does not prove a future runner execution.

The .NET Test Sweep and .NET PR Tests workflows that the inventory credits for most suites are workflow_dispatch-only. They are not an automatic lane.

Commands

A test project was added, moved or withdrawn, or a fixture root moved:

python .gitea/scripts/test/refresh-test-inventory.py
python .gitea/scripts/test/refresh-test-manifest.py
python .gitea/scripts/test/refresh-test-triage.py
python .gitea/scripts/test/refresh-ci-gap-routing.py
python .gitea/scripts/test/refresh-fixture-manifests.py

Then validate and plan:

python .gitea/scripts/test/validate-test-manifest.py --self-test
python .gitea/scripts/test/validate-test-manifest.py \
  --out TestResults/test-manifest-validation/summary.json
python .gitea/scripts/test/validate-ci-routing.py \
  --out TestResults/test-ci-routing/summary.json
python .gitea/scripts/test/plan-test-run.py \
  --changed-file src/Scanner/StellaOps.Scanner.WebService/Program.cs \
  --out-dir TestResults/test-plan-scanner
python .gitea/scripts/test/run-test-plan.py \
  --plan TestResults/test-plan-scanner/test-plan.json \
  --out-dir TestResults/test-run-scanner
python .gitea/scripts/test/plan-test-run.py \
  --select-all \
  --runtime-cost small \
  --suite-shard-total 4 \
  --suite-shard-index 0 \
  --mode nightly \
  --out-dir TestResults/test-plan-nightly-small
python .gitea/scripts/test/summarize-test-readiness.py \
  --summary TestResults/test-run-scanner/test-run-summary.json \
  --out-dir TestResults/test-readiness-scanner \
  --require-executed \
  --report-only
python .gitea/scripts/test/render-test-dashboard.py \
  --plan TestResults/test-plan-scanner/test-plan.json \
  --run-summary TestResults/test-run-scanner/test-run-summary.json \
  --readiness TestResults/test-readiness-scanner/test-readiness-summary.json \
  --out-dir TestResults/test-dashboard-scanner

CI Placement

.gitea/workflows/test-manifest-plan.yml runs on:

• pull requests that change the manifest, testing inventories, planner/validator scripts, or this workflow;
• pushes to main for the same paths;
• manual workflow_dispatch.

The workflow deliberately does not run product-wide suites. It validates manifest drift and runs planner smoke cases for:

• docs-only changes;
• docs sample/template-only changes;
• module code changes;
• shared/global planner behavior through local planner output.

The uploaded artifact is test-manifest-planner under TestResults/.

.gitea/workflows/test-manifest-execution.yml is the shared execution lane:

• pull request and push-to-main runs compute changed files, validate manifest/trait/fixture metadata, build a targeted test-plan.json, execute eligible selected suites/checks, and render a dashboard matrix;
• scheduled nightly/weekly runs use --select-all, shard the active manifest by runtime.cost (small, medium, large), optionally split each runtime shard by suite_shard_total / suite_shard_index, and execute suites whose trigger policy for that mode is not none;
• manual full sweeps use workflow_dispatch with full_sweep=true, mode=manual, execute=true, and bounded runtime_cost / suite_shard_* inputs to select and execute all active suites in controlled chunks;
• manual runs default to dry-run artifacts and can opt into execution with execute=true;
• readiness summaries are produced for every lane. A dry-run summary reports not_evaluated rather than green, and executed lanes fail when selected blocking or required suites, or manifest checks, fail.
• dashboard summaries are printed into job logs and uploaded with test-dashboard.md, test-dashboard.json, and suite-matrix.csv.

CI-gap placement is now explicit in TEST_MANIFEST.yml: the 23 ci-gap rows are registered with targeted PR/main candidate, nightly/manual, weekly/manual, sample-only, extended, or manual-live policy according to runtime and fixture cost. These rows stay report-only until stability evidence promotes them.

See TEST_CI_EVIDENCE_DASHBOARD.md for the evidence standard. A green workflow is not all-suite execution proof unless test-run-summary.json and the dashboard matrix show executed passing rows for the relevant suites.

Current Local Evidence

As of 2026-09-27 (PCD-69, SPRINT_20260927_005):

• manifest coverage: 576 suites and 123 fixture roots; every path exists in the tree and every one of the tracked test projects has a suite row;
• placement: 540 suites run in an automatic lane, 466 of them only in the nightly full sweep (report-only). 36 do not, and each records ci.noneReason: 25 excluded suites and 11 suites that a dedicated workflow runs;
• nightly full-sweep size: 433 suites on the container shards (316 small, 104 medium, 13 large) and 69 on the apparmor-host shard (2 small, 63 medium, 4 large);
• the four validators report 0 errors; validate-test-manifest.py --self-test passes 10 seeded cases (13 since PCD-164).

Before this change the validator compared the manifest only with the inventory CSVs. 107 suite paths and 78 fixture roots named nothing in the tree, 85 test projects had no row, and 496 suites ran in no automatic lane. The nightly sweep logged MSB1009: Project file does not exist for the stale rows and never ran the unregistered projects, among them StellaOps.Vulnerabilities.Ingestion.Tests, which stayed red for 15 days.

As of the first implementation slice:

• manifest coverage: 597 suites and 198 fixture roots;
• validator result: 0 missing suites, 0 missing fixture roots, 0 extra suites, 0 extra fixtures;
• negative validator checks fail as expected for missing suite rows, missing fixture roots, and live E2E PR-blocking policy;
• docs-only planner smoke: 0 selected suites, docs/manifest checks only;
• sample-template planner smoke: 1 selected sample suite;
• Scanner module planner smoke: 79 selected suites;
• shared TestKit planner smoke: 83 selected suites plus nightly follow-up.
• runner dry-run artifacts emit test-run-summary.json and test-run-summary.md from planner output without launching heavy suites.
• command-matrix runner dry-run resolves dotnet, npm/Playwright, Go, and script/tooling command families.
• remote CI run 2827 / #2288 for test-manifest-plan.yml passed on commit 1e8994bebb and published artifact test-manifest-planner with manifest validation, planner smoke, and runner dry-run outputs.
• trait validation covers 147 triaged trait-missing .NET suites and 10 triaged skip-audit suites.
• fixture manifest validation covers Scanner SCM corpus, 52 Concelier connector fixture roots, and 17 cross-cutting src/__Tests/** fixture roots.
• CI routing validation covers all 23 current ci-gap rows and fails if any gap row remains needs-routing or violates production Playwright, benchmark, chaos, or sample placement rules.
• test-manifest-execution.yml now provides targeted PR/main/manual plans plus nightly/weekly/manual full-sweep shards, all using the same planner/runner/readiness/dashboard artifact shape.
• audit note: Gitea run 2888 / #2349 for commit 57eead97ee277eff2adf2a1c46755f2520c0d02d succeeded but selected 0 suites, skipped 598 suites, ran 1 validation check, and skipped all full-sweep shards. Treat that run as planning evidence only, not all-suite execution evidence.