Test CI Evidence Dashboard

Audience: test-automation engineers, release owners, and reviewers auditing Stella Ops CI results. Purpose: define the evidence standard for claiming that manifest-backed test suites actually ran on Gitea CI — not merely that a workflow concluded success.

The core rule: a green test-manifest-execution.yml run is not proof that suites executed. Only executed, passing rows in test-run-summary.json — tied to a workflow path, run ID, commit SHA, and manifest suite ID in the dashboard matrix — count as suite-executed evidence. See the Evidence Standard for the full contract.

Related: TEST_CI_ROUTING.md (trigger placement), TEST_MANIFEST.md (manifest contract), and TEST_FRAMEWORK_CONSOLIDATION_PLAN.md (overall design).

Audit Findings (Run History)

The blocks below capture observed Gitea runs in reverse-chronological order. They are durable evidence of what each run did and did not prove; treat older entries as regression history, not current status.

Initial finding

As of 2026-05-23, a green test-manifest-execution.yml run is not proof that all suites ran.

Initial observed Gitea run:

Conclusion: that run proved manifest validation and planning only. It did not prove suite execution.

Latest all-suite observation:

Conclusion: the manual large sweep proves that runner setup, live progress markers, manual-live blocking, Docker/Testcontainers blocking, benchmark-lane blocking, CLI/Policy NETSDK1188 unblocking, and Signals MSBuild-side NETSDK1188 unblocking work in pushed CI. Treat the blocked suites as missing runner/environment contracts, not passes. The remaining executed failed row is Web AOC DSSE verification: Chrome Headless rejected the SubtleCrypto.verify payload argument as not being a BufferSource. A local WebCrypto normalization fix is implemented and awaits pushed CI proof.

Latest bounded suite-shard observation:

Conclusion: run 3318 closes both sides of medium shard 7/8 on the pushed Tester/EvidenceLocker contract fixes. The default medium job selected/executed 16 suites with 0 failures, and the host-infra medium job selected/executed 4 suites with 0 failures. The earlier shard 7/8 rerun 3317 proved the fixed Tester and EvidenceLocker suites but exposed a report-only Scanner Node transient; local Windows and WSL/Linux runs passed the same Node test project 371/371, and rerun 3318 passed it in CI on stella-runner-1.

Prior bounded run 3317 on commit 4b90d4fe69 proved the fixed tester-stellaops-tester-webservice-tests and evidencelocker-stellaops-evidencelocker-tests rows but exposed one report-only transient in scanner-stellaops-scanner-analyzers-lang-node-tests, which cleared locally on Windows, locally on WSL/Linux, and in rerun 3318. Prior bounded run 3309 on commit 58c545f1ea closed both sides of medium shard 6/8: default job 4541 selected/executed 15 suites with 0 failures, and host-infra job 4544 selected/executed 5 suites with 0 failures, including platform-stellaops-platform-webservice-tests. Prior bounded run 3305 on commit 6fce4d35da closed the default side of medium shard 6/8 and exposed the remaining stale Platform boundary guard in host-infra job 4528. Prior bounded run 3292 on commit 8506293bf7 closed the default side of medium shard 6/8 and exposed the two-failure Platform legacy-alias stale expectation in host-infra job 4495. Prior bounded run 3288 on commit c32cac71bc closed medium shard 5/8 under the split runner contract: default job 4476 selected/executed 15 suites with 0 failures, and host-infra job 4479 selected/executed 5 Docker/Testcontainers suites with 0 failures, including Router Gateway 448/448. Prior bounded run 3283 on commit aff18c550f proved the default side of medium shard 5/8 and proved the MTP referenced-log extraction by naming the DPoP/Valkey setup failure. Prior bounded run 3275 on commit d2acabb3ea proved the default side of medium shard 5/8 and exposed that the dashboard still hid exact MTP failed-test names. Prior bounded run 3271 on commit ef3641041d proved the previous Authority claims-override 404 was gone but exposed fixed local-binding startup noise in the same Router suite. Prior bounded run 3267 on commit 3e533688d3 selected the same fifth medium shard and exposed the now-fixed Authority claims-override 404 in router-stellaops-gateway-webservice-tests. Prior bounded run 3263 closed medium shard 4/8 under the split runner contract: default job 4385 selected/executed 16 suites with 0 failures, and host-infra job 4388 selected/executed 5 Docker/Testcontainers suites with 0 failures. Prior bounded run 3258 on commit baa98dc8ba selected the same fourth medium shard before the runner-placement split and exposed six runner/environment blockers: five Docker/Testcontainers prerequisites plus root package.json missing scripts.test. Prior bounded run 3179 on commit 4598501dbca0ccbd40cae58758888d76c7ca24ef exposed the same six blockers. Prior bounded run 3173 on commit 1efedc042bf2da11c23ec07c9538b6acf801b835 exposed the SmRemote stale/flaky rotation test failure after Graph API passed 99/99. Prior bounded run 3142 on commit 15fae9b1f0daee8ae7665dc9f3d756acde4c3747 selected small shard 3/8 and completed 49 selected, 49 executed, 0 blocked, and 0 failures, closing the Doctor storage restore failure. Prior bounded run 3132 on commit 2c0699b576d35bb995b3a2c3066eecd4cef586bd selected small shard 1/8, completed with 49 selected, 48 executed, 1 blocked, and 0 executed failures, proving the Registry compatibility row blocks before dotnet test and the ReplayableVerdict stale fixture failure no longer appears. Earlier bounded run 3127 on commit af21f09d211750b3eb389c8dd68b6eec748100cb selected the same first shard but produced the two now-classified failures; keep it as regression history, not current shard status.

Latest targeted observation:

Conclusion: this is valid targeted per-commit evidence, not all-suite evidence. It proves a Web source change did not trigger a full suite rerun; the selected Web suite was visible but skipped because its current PR/main CI policy is none.

Evidence Standard

A CI result may be described as suite-executed only when all of these are true:

1. The selected suite has a row in test-run-summary.json.
2. The row has executed: true.
3. The row has status: passed.
4. The row records the command, working directory, trigger reason, mode gating, artifact path, and log path.
5. The dashboard matrix ties the result to workflow path, run ID, commit SHA, event, and manifest suite ID.

A workflow conclusion of success without executed suite rows is planning evidence only.

Dashboard Artifacts

render-test-dashboard.py joins:

• docs/technical/testing/TEST_MANIFEST.yml
• test-plan.json
• test-run-summary.json
• test-readiness-summary.json

and emits:

Every matrix row includes:

• suite ID, owner, category, repo path, runner, runtime cost, estimated tests, and infra;
• layers, scenario families, E2E scenario count/examples, seed data, and fixtures;
• PR/main/nightly/weekly/manual/gating policy;
• whether the suite was selected, why it was selected, and its mode gating;
• run status, executed/dry-run flags, skipped reason, command, working directory, artifact/log path, exit code, error, start/completion timestamps, duration, and failure tail.

The workflow prints the complete Markdown dashboard to logs because artifact download or artifact API visibility can fail independently of the job result. When failures or blocked prerequisites are present, the Markdown dashboard also includes a Failure Output Tails section so an outside reviewer can see more than a one-line error preview.

The selected suite matrix in test-dashboard.md must be useful on its own when artifacts are unavailable. It therefore includes suite name, category/layers/scenario families, seed and fixture references, runner, working directory or repo path, gating mode, status, trigger reason, evidence path, and an Executed Suite Commands table for the exact commands that ran.

Failure Diagnostics

The dashboard must expose the useful diagnostic line, not the first line emitted by a tool. Some suites print benign build or bundler output before the real assertion. For example, run 3000 showed the Web suite failure row as Angular initial chunk output, which hid the actual assertion when the artifact API returned no artifacts.

run-test-plan.py therefore prints a live result line for failed suites with:

[test-run] 2026-05-24T00:00:00Z suite 1/42 web-stellaops-web end status=failed exit_code=1 duration_seconds=123.456 log=TestResults/... failure_detail="..."

render-test-dashboard.py also scans the suite output tail for diagnostic markers such as error, failed, exception, assertion, expected, actual, NETSDK, TypeError, and ReferenceError. The selected diagnostic line appears in the failure table, and the Failure Output Tails section prints the end of the log where assertions and stack traces usually appear.

For .NET suites using Microsoft Testing Platform, the actionable failure can live outside the main suite stdout in a referenced TestResults/*.log file. run-test-plan.py therefore parses TRX files first, then discovers MTP referenced logs from Tests failed: '...' output and from the executed .csproj bin/**/TestResults/*.log tree. It appends a failure-focused extract from those logs to the suite log so the job-log dashboard can show failed test names, messages, expected/actual values, and stack heads even when artifacts are not available.

The Gitea workflow prints the full dashboard Markdown to the job log. This keeps the durable log path aligned with the JSON/CSV artifacts so an outside reviewer can identify all selected suites, why they were selected or blocked, where they ran, and the command used for executed rows even when artifact download is unavailable.

Run 3006 proved the larger tail is useful: the Web row no longer hid behind Angular chunk tables and instead exposed Angular test process terminated by signal SIGKILL plus Batch 1/35 failed, while the Signals row exposed the remaining NETSDK1188 package-resource warning path.

Run 3031 proved the current diagnostic path is actionable for browser assertions. The Web failure table named src/app/core/aoc/provenance-builder.spec.ts, the failure tail named verifies DSSE signatures using pre-auth encoding, and the error body showed:

TypeError: Failed to execute 'verify' on 'SubtleCrypto': 4th argument is not instance of ArrayBuffer, Buffer, TypedArray, or DataView.

The same job-log dashboard printed the batch scope and focused rerun command, so an outside reviewer can see both what failed and how to reproduce the selected batch.

Run 3038 then proved there is a separate Web runner-resource condition: the first Angular batch was terminated by SIGKILL before assertion output. The Web batch runner therefore uses CI-specific defaults of 6 files per batch and --max-old-space-size=2048, while local developer runs keep 12 files and 3072 MB unless overridden by STELLAOPS_WEB_TEST_BATCH_SIZE or STELLAOPS_WEB_TEST_NODE_MEMORY_MB.

Run 3098 exposed a medium-shard runner precondition instead of a stale property test: tests-stellaops-testing-determinism-properties failed Unicode equivalence checks because the workflow set DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1. Active .NET workflows now set that value to 0, and manifest suites that require Unicode NFC normalization can declare dotnet-full-globalization so the runner blocks with a clear prerequisite message if invariant globalization is reintroduced.

For the Web Angular unit batch runner, a failed batch must also print the included spec file paths and a focused rerun command:

Batch 1/35 failed. Included spec files:
  - src/app/app.component.spec.ts
Rerun this batch with: npm run test -- --batch-from 1 --batch-to 1

This gives reviewers a concrete failing scope even when the underlying browser-test output has been truncated or reduced to a generic batch failure line.

Live Progress Logs

Long full-sweep shards must print progress before the final dashboard, because an outsider should be able to tell which suite is currently consuming time. run-test-plan.py emits lines like:

[test-run] 2026-05-23T18:00:00Z suite 12/389 scanner-storage-tests start gating=report-only cwd=. log=TestResults/... command="dotnet test ..."
[test-run] 2026-05-23T18:03:10Z suite 12/389 scanner-storage-tests end status=passed exit_code=0 duration_seconds=190.123 log=TestResults/...

The final dashboard still remains the authoritative pass/fail matrix, but live markers make long shards diagnosable before the shard completes.

Suite stdout/stderr is still captured to per-suite logs, but run-test-plan.py also streams selected high-signal lines with a [test-run-output] prefix. This keeps long Web batches and assertion failures visible while avoiding a full log dump for every suite.

Safe Live Observation

Do not poll full active job logs while a manual full sweep is still running. Gitea serves the whole job log for actions/jobs/<id>/logs; repeated downloads of growing Web/.NET logs add avoidable load to the same host that is running the CI containers and Gitea.

Use this contract instead:

• While jobs are active, poll run/job status only, with at least 30 seconds between checks.
• Use the Gitea UI for spot checks when needed instead of scripted full-log loops.
• Fetch job logs after a job reaches a terminal state, or make one explicit active-log pull for a specific diagnosis.
• Do not run multiple local pollers against the same run.
• If Gitea or the host network becomes unstable, stop host-level investigation from the test-automation thread. Record the incident in the sprint and continue only with local repository work plus low-rate Gitea status/API reads until DevOps completes the host review.

The history collector enforces the safe default: --include-logs skips active job logs unless --include-running-logs is explicitly supplied.

Runner Setup Evidence

detect-plan-runner-needs.py reads the generated plan before execution and emits per-shard setup requirements:

• whether any runnable suite needs .NET, Go, Node/npm, or Playwright;
• the count of runnable selected suites;
• npm package roots that require npm ci.

test-manifest-execution.yml uses those outputs to provision only the toolchains required by the suites selected for that commit or runtime shard. A docs-only or manifest-only change with zero selected runnable suites must not force broad .NET/Go/Node setup.

.NET suite commands are intentionally constrained for self-hosted runner stability. run-test-plan.py appends:

--disable-build-servers /m:1 /p:BuildInParallel=false /p:UseSharedCompilation=false /p:RestoreDisableParallel=true /nodeReuse:false

These flags are not proof that a product test passed. They are a runner contract so CI failures expose real build/test output instead of MSBuild/Roslyn resource pressure. If this still fails before test execution, the suite remains a runner/toolchain blocker until the full per-suite log identifies a product assertion failure.

Runner Capacity Contract

The default Gitea runner fleet is intentionally capped for host stability:

• only runner-1 and runner-2 should be online for normal CI and manual full-sweep evidence;
• runner-3 is standby capacity in the Docker Compose standby profile;
• each runner has runner.capacity: 1;
• nested job containers are limited to --cpus=2 --memory=3g;
• runner polling is 5s;
• the host has a 4 GiB swap file at /data/swapfile-gitea-ci with vm.swappiness=10.

A manual full sweep may take longer under this cap, but it must not make the Gitea UI/API unavailable. If an operator temporarily starts runner-3, they must not also run full active-log polling loops, and they should stop runner-3 after the controlled workload.

Full-Sweep Suite Sharding

Runtime shards (small, medium, large) are not always small enough for a constrained self-hosted runner. Manual and scheduled full-sweep lanes therefore support an additional deterministic suite shard inside each runtime shard:

• suite_shard_total: number of suite shards to split the selected runtime shard into;
• suite_shard_index: zero-based shard index to run, from 0 through suite_shard_total - 1.

The planner first filters active suites by runtime_cost, then keeps suites whose manifest-order ordinal belongs to the requested suite shard. Dashboards print both Runtime shard and Suite shard, and test-plan.json records how many suites were selected before and after suite sharding.

Example controlled manual dispatch for the first quarter of the small runtime shard:

mode=manual
full_sweep=true
execute=true
runtime_cost=small
suite_shard_total=4
suite_shard_index=0

All-suite evidence is complete only when every required runtime shard and every required suite shard has terminal dashboard evidence. A green small shard 1/4 is not proof for the other three small shards.

Manual-Live Prerequisites

Production/live Playwright suites that declare live Stella Ops stack in the manifest must not fail with an opaque browser/auth stack when the runner lacks a live environment. run-test-plan.py blocks those suites before execution unless these values are present:

• STELLA_BASE_URL
• one username source: STELLA_USERNAME, STELLAOPS_FRONTDOOR_USERNAME, or STELLAOPS_ADMIN_USER
• one password source: STELLA_PASSWORD, STELLAOPS_FRONTDOOR_PASSWORD, or STELLAOPS_ADMIN_PASS

The result status is blocked, not passed; readiness treats blocked required suites as blocking failures. This makes the missing environment visible without hiding the fact that the live suite still has no pass evidence.

Docker/Testcontainers Prerequisites

Suites that declare docker-testcontainers or testcontainers in the manifest require a runner with a usable Docker daemon before their failures can be treated as product evidence. run-test-plan.py checks Docker once per shard with:

docker info --format "{{.ServerVersion}}"

If Docker is missing, unhealthy, or times out, the suite result is blocked, the summary increments both blockedSuiteCount and failureCount, and the suite log records the runner prerequisite failure. A runner can bypass the preflight only when a controlled equivalent is provided and STELLAOPS_TESTCONTAINERS_AVAILABLE=true is set by the workflow or operator.

This is not a skip or a pass. It prevents Docker/Testcontainers environment gaps from being confused with product assertions.

Docker/Testcontainers rows must also avoid depending on live public registry behavior when the required image can be provided by CI. The host-infra manifest job restores a Docker image cache and resolves DockerHub images through CI_DOCKERHUB_MIRROR_PREFIX when configured. Suites that need a shared image should honor workflow-provided environment variables such as STELLAOPS_TEST_VALKEY_IMAGE before falling back to public tags.

.NET Full Globalization Prerequisites

Canonical JSON and Unicode determinism tests require .NET full globalization/ICU support. Workflows must not set:

DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1

Suites tagged with dotnet-full-globalization are blocked before execution when invariant globalization is enabled. The dashboard records status=blocked and the missing prerequisite instead of showing an opaque Unicode string diff.

BenchmarkDotNet Prerequisites

Pure BenchmarkDotNet rows are not normal xUnit/MTP suites. run-test-plan.py blocks manifest rows that are benchmark-category/layer/infra rows with zero estimated tests before trying dotnet test.

The required fix is one of:

• add a distinct weekly/manual dotnet/benchmarkdotnet runner with an executable benchmark entry point, timeout, and resource budget;
• or add a separate small xUnit smoke project for PR/main correctness and keep the benchmark project in the benchmark lane.

This preserves benchmark visibility without creating misleading dotnet test failures for projects that do not contain executable tests.

Trigger Interpretation

Concurrency Policy

test-manifest-execution.yml uses a concurrency group that includes both ref and event class:

concurrency:
  group: test-manifest-execution-${{ github.ref }}-${{ github.event_name }}
  cancel-in-progress: true

This keeps newer push checks from wasting runner time on stale push checks, while preventing push churn from cancelling a manually dispatched all-suite proof run. A newer manual dispatch for the same ref still cancels an older manual dispatch.

Historical Run Collection

Use collect-gitea-test-history.py to map Gitea runs to commits and job results without storing credentials in the repository.

export GITEA_TOKEN=...
python .gitea/scripts/test/collect-gitea-test-history.py \
  --limit 50 \
  --include-logs \
  --out-dir TestResults/gitea-test-history

By default, --include-logs only downloads logs for jobs that are no longer active. To deliberately inspect an in-progress job log, pass --include-running-logs; do this sparingly and only for a specific diagnosis.

The collector writes:

• gitea-test-history.json
• gitea-test-history.csv
• gitea-test-history.md

The history collector records workflow path, run ID, event, conclusion, commit SHA, run number, job names, job conclusions, runner names, and whether dashboard/execution-count markers were visible in job logs.

When --include-logs is enabled, the collector also parses the printed dashboard summary from each job log and carries the key run-matrix values into the history JSON/CSV/Markdown:

• dashboard label, readiness state, and runtime shard;
• suite shard identity when present;
• selected, executed, blocked, total failure, and blocking-suite counts;
• failure suite IDs, statuses, and compact diagnostic details from the dashboard failure table and Failure Output Tails section.

This makes the history output useful as a commit-to-suite-status index instead of only a list of workflow conclusions.

All-Suite Proof Status (2026-05-29 closure)

The manifest currently contains 597 suites: 594 active and 3 sample. The current CI policy is intentionally conservative:

• PR/main: most suites are none; only related targeted/candidate/blocking rows are eligible.
• Nightly: nightly full sweep on test-manifest-execution.yml (cron: 17 1 * * *) selects every active suite under --select-all and splits by infra. Three consecutive nightly schedules (run_number 2787 on c030c594 for 2026-05-27, run_number 2865 on 090f498b for 2026-05-28, run_number 3136 on 8d3f64ed for 2026-05-29) all completed success on every shard (Full manifest sweep small/medium/large + Full manifest sweep host infra small/medium/large). The targeted lane correctly skips for schedule events.
• Manual: all active suites have required, allowed, or report-only manual policy and can also be selected through manual full-sweep runtime shards.

The 594 active suites partition exactly into 6 lanes with zero overlap and zero gap:

The two gating=manual-live suites (web-stellaops-web live Playwright portion and tests-playwright-operator-trust-playwright) are documented blockers per SPRINT_006_PER_SUITE_EVIDENCE.md. They remain manual-live until a controlled operator runner with STELLA_BASE_URL/credentials is provisioned and a manual mode=manual full_sweep=true runtime_cost=large execute=true dispatch is run there.

Co-requisite scheduled lanes also green on the same three consecutive nights:

• e2e-notify-coverage.yml Full 4-phase run (mock receivers): runs 2763, 2790, 2868, 3139 all success.
• workflow-renderer.yml Renderer fast lane + Explicit slow renders (24-node DocumentProcessingWorkflow): runs 2786, 2796, 2898, 3195 all success.
• airgap-federation-e2e.yml Tier 2 live docker e2e: runs 2764, 2791, 2869, 3140 all success.

See docs/technical/testing/SPRINT_006_PER_SUITE_EVIDENCE.md and SPRINT_006_PER_SUITE_EVIDENCE.csv for the per-suite matrix.