Test CI Routing

Audience: CI/release owners and engineers who need to know when and where a given change runs tests. Purpose: define the trigger placement for the manifest-backed testing framework.

The contract is intentionally conservative: every commit is analyzed, but broad/full execution is restricted to nightly, weekly, or manual lanes. This keeps PR feedback fast and CI cost bounded while still giving expensive, live, and Docker-heavy suites a visible home.

For the evidence standard that governs when a routed suite counts as “executed,” see TEST_CI_EVIDENCE_DASHBOARD.md. For the manifest contract behind these triggers, see TEST_MANIFEST.md.

Trigger Classes

Trigger class	Workflow placement	Contract
PR / per-commit	`.gitea/workflows/test-manifest-execution.yml` targeted job	Build `test-plan.json` from changed files and execute eligible selected suites/checks. Report-only and candidate failures stay visible without becoming release blockers.
Merge to main	`.gitea/workflows/test-manifest-execution.yml` targeted job	Re-plan from the last commit and publish the same plan/run/readiness/dashboard artifacts for main.
Nightly	`.gitea/workflows/test-manifest-execution.yml` full-sweep jobs, daily cron	Run runtime shards and execute suites whose `ci.nightly` policy is not `none`; Docker/Testcontainers rows are routed to the `apparmor-host` host-infra job while default shards exclude them.
Weekly / release	`.gitea/workflows/test-manifest-execution.yml` full-sweep jobs, weekly cron or `workflow_dispatch mode=weekly`	Keep benchmark, chaos, and extended suites visible without running them on unrelated PRs; execute non-`none` weekly suites with the same default-runner vs host-infra split.
Manual full sweep	`.gitea/workflows/test-manifest-execution.yml` with `workflow_dispatch full_sweep=true mode=manual execute=true`	Select every active suite by runtime shard, split into default non-Docker shards and `apparmor-host` Docker/Testcontainers shards. This is the current all-active-suite execution lane.
Manual / live	`.gitea/workflows/test-manifest-execution.yml` with `workflow_dispatch`	Operators can set `execute=true` for selected lanes. Live/operator rows stay manual-first unless explicitly promoted.
Manifest drift	`.gitea/workflows/test-manifest-plan.yml`	Validate manifest, trait, fixture, and CI routing metadata plus planner/runner dry-run artifacts.

Current CI-Gap Placement

test-ci-gap-routing.csv is generated from TEST_MANIFEST.yml and covers the 23 current ci-gap suites.

Family	Placement
Production/operator Playwright (`src/__Tests/playwright/**`)	Nightly/manual visible, `manual-live` gating, no PR/main execution by default.
`src/__Tests/e2e/**`	Related-change PR/main candidate, report-only until stability evidence exists.
`src/__Tests/Integration/**`	Related-change PR/main candidate, report-only until stability evidence exists.
`src/Doctor/__Tests/**`	Related-change PR/main candidate, report-only until stability evidence exists.
BinaryIndex and performance benchmarks	Weekly/manual only, never PR.
Docs samples/templates	Sample-only targeted PR/main, never product-wide.

No Over-Trigger Rules

Docs-only changes produce docs/manifest checks, not product-wide suites.
Docs sample/template changes select only sample/template suites.
Fixture-only changes select suites that consume that fixture root.
Shared/global changes select a bounded representative PR slice and add a nightly follow-up check.
Manifest tooling changes under docs/technical/testing, .gitea/scripts/test, or manifest workflows run manifest/trait/fixture/routing checks only; they do not select product suites by themselves.
--select-all is reserved for nightly, weekly, and manual sweeps.

Runner Placement

Default full-sweep jobs run on ubuntu-latest, which is a nested container runner without Docker CLI/socket access. These jobs pass --exclude-infra docker-testcontainers --exclude-infra testcontainers.
Docker/Testcontainers full-sweep jobs run on apparmor-host, where the runner contract includes a working host Docker daemon. These jobs pass --include-infra docker-testcontainers --include-infra testcontainers.
The suite shard ordinal remains the same across the split. For example, medium 4/8 now produces a non-Docker shard and a host-infra shard that together cover the same rows previously selected by the unsplit medium 4/8 plan.
detect-plan-runner-needs.py provisions setup from the selected plan. Package rows that declare dotnet-backed guard scripts in runtime.infra request .NET setup even when their runner is npm/node.

Release Readiness

summarize-test-readiness.py reads test-run-summary.json and writes:

test-readiness-summary.json
test-readiness-summary.md

Readiness cannot be green when selected blocking or required suites are failed, skipped, cancelled, or timed out. Dry-run summaries become not_evaluated when --require-executed is set.

render-test-dashboard.py also writes the per-suite evidence matrix:

test-dashboard.md
test-dashboard.json
suite-matrix.csv

The Markdown dashboard is printed into CI logs so an outside reviewer can see selected suites, run reasons, execution counts, skipped-by-policy counts, and failure evidence without relying only on artifact download.

Commands

python .gitea/scripts/test/refresh-ci-gap-routing.py
python .gitea/scripts/test/validate-ci-routing.py \
  --out TestResults/test-ci-routing/summary.json

python .gitea/scripts/test/plan-test-run.py \
  --changed-file src/Scanner/StellaOps.Scanner.WebService/Program.cs \
  --mode pr \
  --out-dir TestResults/test-plan-targeted

python .gitea/scripts/test/plan-test-run.py \
  --select-all \
  --runtime-cost small \
  --exclude-infra docker-testcontainers \
  --exclude-infra testcontainers \
  --mode nightly \
  --out-dir TestResults/test-plan-nightly-small

python .gitea/scripts/test/plan-test-run.py \
  --select-all \
  --runtime-cost small \
  --include-infra docker-testcontainers \
  --include-infra testcontainers \
  --mode nightly \
  --out-dir TestResults/test-plan-nightly-small-host-infra

python .gitea/scripts/test/run-test-plan.py \
  --plan TestResults/test-plan-targeted/test-plan.json \
  --out-dir TestResults/test-run-targeted \
  --dry-run \
  --report-only

python .gitea/scripts/test/summarize-test-readiness.py \
  --summary TestResults/test-run-targeted/test-run-summary.json \
  --out-dir TestResults/test-readiness-targeted \
  --require-executed \
  --report-only

python .gitea/scripts/test/render-test-dashboard.py \
  --plan TestResults/test-plan-targeted/test-plan.json \
  --run-summary TestResults/test-run-targeted/test-run-summary.json \
  --readiness TestResults/test-readiness-targeted/test-readiness-summary.json \
  --out-dir TestResults/test-dashboard-targeted