Test Manifest

Audience: test-automation engineers and anyone wiring a suite into Stella Ops CI. Purpose: document the implemented test-manifest contract — its files, the required suite and fixture fields, the planner/validator/runner commands, and how the manifest drives CI placement.

The test manifest is the consolidation control point for suite ownership, fixture ownership, trigger routing, and CI gating. It is the source of truth that the change-impact planner and CI lanes consume; see TEST_FRAMEWORK_CONSOLIDATION_PLAN.md for the design rationale and TEST_CI_ROUTING.md for trigger placement.

Files

TEST_MANIFEST.yml - generated manifest. It uses JSON syntax inside a .yml file so CI can parse it with the Python standard library only.
TEST_MANIFEST.schema.json - suite and manifest field contract.
FIXTURE_MANIFEST.schema.json - fixture entry contract for future split fixture manifests.
.gitea/scripts/test/refresh-test-manifest.py - regenerates the manifest from the suite, fixture, and E2E inventory CSV snapshots.
.gitea/scripts/test/validate-test-manifest.py - validates manifest coverage and invalid trigger/gating combinations.
.gitea/scripts/test/plan-test-run.py - creates test-plan.json and test-plan.md for a set of changed files.
.gitea/scripts/test/run-test-plan.py - consumes test-plan.json and emits stable runner summaries. It is dry-run by default.
.gitea/scripts/test/summarize-test-readiness.py - turns a runner summary into a release-readiness rollup and fails when selected blocking/required suites or validation checks failed.
.gitea/scripts/test/render-test-dashboard.py - joins manifest, plan, run, and readiness artifacts into a Markdown/JSON/CSV execution matrix.
.gitea/scripts/test/collect-gitea-test-history.py - uses GITEA_TOKEN from the environment to collect recent Gitea run/job history without storing credentials.
.gitea/scripts/test/refresh-ci-gap-routing.py - regenerates the explicit CI-gap routing table from the manifest.
.gitea/scripts/test/validate-ci-routing.py - validates CI-gap trigger placement and no-overtrigger constraints.
.gitea/workflows/test-manifest-plan.yml - CI validation workflow for manifest and planner changes.
.gitea/workflows/test-manifest-execution.yml - shared PR/main/nightly/weekly/manual entry point that consumes the planner and runner artifacts.

Required Suite Fields

Every runnable suite row must be represented or explicitly excluded.

Field	Purpose
`id`	Stable suite identifier used by CI and result artifacts.
`status`	`active`, `sample`, or `excluded`. Sample suites are runnable examples and must not trigger product-wide suites.
`owner`	Owning module or test surface.
`name`	Project/package/suite name.
`path`	Source path for the suite.
`runner`	Runner family such as `dotnet test`, `npm/playwright`, `go test`, or script/tooling.
`category`	Primary category used for CI routing and .NET trait migration.
`layers`	Secondary observed layers from inventory, such as unit, integration, e2e, contract, golden, or determinism.
`ci`	Per-trigger policy for PR, main, nightly, weekly, manual, and overall gating state.
`triggerPaths`	Paths that should select the suite in the change-impact planner.
`runtime`	Cost, estimated tests/spec files, and required infrastructure.
`fixtures` / `fixtureRoots`	Fixture roots consumed by the suite.
`seedData`	Seed-data hints from project inventory.
`traits`	Current `Category`, `Lane`, and other static trait signals.
`inventory`	Source inventory metadata and consolidation recommendation.

Required Fixture Fields

Every fixture or corpus root must be represented or explicitly excluded.

Field	Purpose
`id`	Stable fixture identifier.
`owner`	Owning module, doc area, or shared test surface.
`path`	Fixture root path.
`corpusType`	Fixture, seed data, golden files, corpus, or generated copy classification.
`fileCount`	Inventory file count.
`sizeBytes`	Inventory byte size.
`extensions`	Observed file extensions.
`likelyGeneratedCopy`	Marks generated fixture copies that should not become source-of-truth seed data.
`status`	`active` or `excluded`.

Commands

python .gitea/scripts/test/refresh-test-manifest.py
python .gitea/scripts/test/validate-test-manifest.py \
  --out TestResults/test-manifest-validation/summary.json
python .gitea/scripts/test/validate-ci-routing.py \
  --out TestResults/test-ci-routing/summary.json
python .gitea/scripts/test/plan-test-run.py \
  --changed-file src/Scanner/StellaOps.Scanner.WebService/Program.cs \
  --out-dir TestResults/test-plan-scanner
python .gitea/scripts/test/run-test-plan.py \
  --plan TestResults/test-plan-scanner/test-plan.json \
  --out-dir TestResults/test-run-scanner
python .gitea/scripts/test/plan-test-run.py \
  --select-all \
  --runtime-cost small \
  --suite-shard-total 4 \
  --suite-shard-index 0 \
  --mode nightly \
  --out-dir TestResults/test-plan-nightly-small
python .gitea/scripts/test/summarize-test-readiness.py \
  --summary TestResults/test-run-scanner/test-run-summary.json \
  --out-dir TestResults/test-readiness-scanner \
  --require-executed \
  --report-only
python .gitea/scripts/test/render-test-dashboard.py \
  --plan TestResults/test-plan-scanner/test-plan.json \
  --run-summary TestResults/test-run-scanner/test-run-summary.json \
  --readiness TestResults/test-readiness-scanner/test-readiness-summary.json \
  --out-dir TestResults/test-dashboard-scanner

CI Placement

.gitea/workflows/test-manifest-plan.yml runs on:

pull requests that change the manifest, testing inventories, planner/validator scripts, or this workflow;
pushes to main for the same paths;
manual workflow_dispatch.

The workflow deliberately does not run product-wide suites. It validates manifest drift and runs planner smoke cases for:

docs-only changes;
docs sample/template-only changes;
module code changes;
shared/global planner behavior through local planner output.

The uploaded artifact is test-manifest-planner under TestResults/.

.gitea/workflows/test-manifest-execution.yml is the shared execution lane:

pull request and push-to-main runs compute changed files, validate manifest/trait/fixture metadata, build a targeted test-plan.json, execute eligible selected suites/checks, and render a dashboard matrix;
scheduled nightly/weekly runs use --select-all, shard the active manifest by runtime.cost (small, medium, large), optionally split each runtime shard by suite_shard_total / suite_shard_index, and execute suites whose trigger policy for that mode is not none;
manual full sweeps use workflow_dispatch with full_sweep=true, mode=manual, execute=true, and bounded runtime_cost / suite_shard_* inputs to select and execute all active suites in controlled chunks;
manual runs default to dry-run artifacts and can opt into execution with execute=true;
readiness summaries are produced for every lane. A dry-run summary reports not_evaluated rather than green, and executed lanes fail when selected blocking or required suites, or manifest checks, fail.
dashboard summaries are printed into job logs and uploaded with test-dashboard.md, test-dashboard.json, and suite-matrix.csv.

CI-gap placement is now explicit in TEST_MANIFEST.yml: the 23 ci-gap rows are registered with targeted PR/main candidate, nightly/manual, weekly/manual, sample-only, extended, or manual-live policy according to runtime and fixture cost. These rows stay report-only until stability evidence promotes them.

See TEST_CI_EVIDENCE_DASHBOARD.md for the evidence standard. A green workflow is not all-suite execution proof unless test-run-summary.json and the dashboard matrix show executed passing rows for the relevant suites.

Current Local Evidence

As of the first implementation slice:

manifest coverage: 597 suites and 198 fixture roots;
validator result: 0 missing suites, 0 missing fixture roots, 0 extra suites, 0 extra fixtures;
negative validator checks fail as expected for missing suite rows, missing fixture roots, and live E2E PR-blocking policy;
docs-only planner smoke: 0 selected suites, docs/manifest checks only;
sample-template planner smoke: 1 selected sample suite;
Scanner module planner smoke: 79 selected suites;
shared TestKit planner smoke: 83 selected suites plus nightly follow-up.
runner dry-run artifacts emit test-run-summary.json and test-run-summary.md from planner output without launching heavy suites.
command-matrix runner dry-run resolves dotnet, npm/Playwright, Go, and script/tooling command families.
remote CI run 2827 / #2288 for test-manifest-plan.yml passed on commit 1e8994bebb and published artifact test-manifest-planner with manifest validation, planner smoke, and runner dry-run outputs.
trait validation covers 147 triaged trait-missing .NET suites and 10 triaged skip-audit suites.
fixture manifest validation covers Scanner SCM corpus, 52 Concelier connector fixture roots, and 17 cross-cutting src/__Tests/** fixture roots.
CI routing validation covers all 23 current ci-gap rows and fails if any gap row remains needs-routing or violates production Playwright, benchmark, chaos, or sample placement rules.
test-manifest-execution.yml now provides targeted PR/main/manual plans plus nightly/weekly/manual full-sweep shards, all using the same planner/runner/readiness/dashboard artifact shape.
audit note: Gitea run 2888 / #2349 for commit 57eead97ee277eff2adf2a1c46755f2520c0d02d succeeded but selected 0 suites, skipped 598 suites, ran 1 validation check, and skipped all full-sweep shards. Treat that run as planning evidence only, not all-suite execution evidence.