Test Failure Triage - 2026-05-23

Scope

Source CI run: Gitea Actions run 2938, workflow test-manifest-execution.yml, manual full sweep, commit 157a31f5a5df51ffb3edd5fc3bcb1b07c3edc566.
Evidence path: job log dashboard from large shard job 3759; Gitea artifact API returned zero artifacts for the run, so the job log is the current durable evidence source.
Large shard result: 42 selected, 42 executed, 33 passed, 9 failed, 2 blocking.
Medium shard job 3760 and small shard job 3761 were still in_progress at 2026-05-23 19:48 UTC.

Run 3000 Refresh

Latest checked CI run: Gitea Actions run 3000, workflow test-manifest-execution.yml, manual large sweep, commit e956b656e5621d379fe99b8a283be3136c0b0424.
Evidence path: job log dashboard from large shard job 3863; Gitea artifact API again returned total_count: 0, so job-log dashboards and live per-suite lines remain the durable evidence path.
Large shard result: 42 selected, 32 executed, 10 blocked, 14 failures, 4 blocking failed suites.
Blocked suites: 8 Docker/Testcontainers prerequisite blockers, 1 manual-live operator prerequisite blocker, and 1 pure BenchmarkDotNet runner-contract blocker.
Remaining failed suites: web-stellaops-web, cli-stellaops-cli-tests, policy-stellaops-policy-tests, and scanner-stellaops-scanner-webservice-tests.
Current classification: no production behavior bug is confirmed from run 3000. CLI and Policy failed before meaningful assertions on NETSDK1188; Web failed with insufficient visible diagnostic detail; Scanner WebService needed corrected manifest infra so Docker/Testcontainers prerequisites block before the test host starts.
Implemented local fixes after run 3000: diagnostic failure-detail extraction, longer job-log dashboard preview, central non-fatal NETSDK1188 handling, Scanner WebService Testcontainers/Postgres manifest infra replacing the misleading BenchmarkDotNet package signal, and two stale CLI test repairs found by the adjacent spec run.

Run 3006 Refresh

Latest checked manual-large CI run: Gitea Actions run 3006, workflow test-manifest-execution.yml, commit 1f21c8bc8903f1b13583ff61b519912aa07ae4fa.
Evidence path: job log dashboard from large shard job 3875; dashboard artifacts were uploaded, but the job-log dashboard remains the durable outside-review path.
Large shard result: 42 selected, 31 executed, 29 passed, 11 blocked, 2 failed, 13 failure/blocker rows, and 4 blocking suites.
Blocked suites: 9 Docker/Testcontainers prerequisite blockers, 1 manual-live operator prerequisite blocker, and 1 pure BenchmarkDotNet runner-contract blocker.
Newly proven green from the previous failure pool: cli-stellaops-cli-tests, policy-stellaops-policy-tests, policy-stellaops-policy-engine-tests, and scanner-stellaops-scanner-reachability-tests.
Remaining actual failed suites: web-stellaops-web and signals-stellaops-signals-tests.
Current classification: Web is now a CI-only termination/resource/logging problem until the fresh batch-scope logging commit is exercised (SIGKILL before assertions in CI, but local batch 1 passed). Signals is still NETSDK1188, but through the MSBuild package-resource path; local MSBuildWarningsAsMessages=NETSDK1188 validation passed and requires pushed CI proof.

Run 3031 Refresh

Latest checked manual-large CI run: Gitea Actions run 3031, workflow test-manifest-execution.yml, commit 745a96eb93a77a8c025164974c2a7b037d441374.
Evidence path: job log dashboard from large shard job 3924; medium and small shard jobs completed successfully with empty shard semantics because the dispatch requested runtime_cost=large.
Large shard result: 42 selected, 31 executed, 30 passed, 11 blocked, 1 failed, 12 failure/blocker rows, and 4 blocking suites.
Blocked suites: 9 Docker/Testcontainers prerequisite blockers, 1 manual-live operator prerequisite blocker, and 1 pure BenchmarkDotNet runner-contract blocker.
Newly proven green from the previous failure pool: signals-stellaops-signals-tests passed in pushed CI (44.475s), proving the MSBuild-side NETSDK1188 demotion.
Remaining actual failed suite: web-stellaops-web.
Current classification: Web is now a confirmed browser/runtime BufferSource bug in the AOC DSSE verification path, not a generic SIGKILL or logging problem. The failing CI assertion is verifies DSSE signatures using pre-auth encoding; Chrome Headless rejects the payload passed to SubtleCrypto.verify as not being an ArrayBuffer/typed-array BufferSource. Local fix normalizes typed-array views structurally and copies WebCrypto inputs into fresh ArrayBuffer instances before verify/digest; pushed CI proof is still required.

Run 3127 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3127, workflow test-manifest-execution.yml, commit af21f09d211750b3eb389c8dd68b6eec748100cb.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=small, suite_shard_total=8, suite_shard_index=0.
Evidence path: job log dashboard from small shard job 4085, plus refreshed history snapshot at TestResults/gitea-test-history-3127/.
Small shard result: 49 selected, 49 executed, 47 passed, 0 blocked, 2 failed, readiness ready because the failures were not blocking readiness gates.
Empty shard jobs: large and medium completed successfully with 0 selected because the dispatch requested runtime_cost=small.
Suite-shard proof: the Gitea dashboard printed Runtime shard: small and Suite shard: 1/8; the history collector parsed full-sweep-small:small/1/8.
Artifact upload note: the small shard upload initially received Gitea/nginx 429, retried, then finalized successfully with 66 files uploaded.
Post-run host health: external Gitea health passed, airgap_containers=0, actions_networks=0, active_job_containers=0, and host memory was stable after the job completed.
Failure group 1: tests-stellaops-infrastructure-registry-testing-tests failed in RegistryCompatibilityFixture.InitializeAsync() with $XunitDynamicSkip$Registry compatibility tests require Docker. No registry containers could be started. Current classification is runner-contract/manifest infra misclassification: this suite should be blocked before execution on Dockerless runners.
Failure group 2: tests-stellaops-e2e-replayableverdict failed 6/12 tests with deterministic verdict hash mismatch. FullPipeline_RequiresIntegration expected gs:sha256:39c4d35cfd1055b01dfab51d5a22516335c6c07e, but actual was gs:sha256:57e38a5bfc0734cd6a799f5591412ea537d8329e. Current classification after local reproduction is stale golden fixture data, not a verdict library regression; both replay and direct build compute the newer 57e38... CGS hash deterministically.

Run 3132 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3132, workflow test-manifest-execution.yml, commit 2c0699b576d35bb995b3a2c3066eecd4cef586bd.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=small, suite_shard_total=8, suite_shard_index=0.
Evidence path: job log dashboard from small shard job 4096, plus refreshed history snapshot at TestResults/gitea-test-history-3132/.
Small shard result: 49 selected, 48 executed, 1 blocked, 0 executed failures, readiness ready.
Empty shard jobs: large and medium completed successfully with 0 selected because the dispatch requested runtime_cost=small.
Registry compatibility result: tests-stellaops-infrastructure-registry-testing-tests is now status=blocked before execution on this Dockerless runner. This confirms the manifest infra correction and keeps Docker/Testcontainers/registry work as an explicit runner-contract blocker.
ReplayableVerdict result: tests-stellaops-e2e-replayableverdict no longer appears in failure details on the same bounded shard after the fixture hash update and output-hash guard. This closes the stale golden fixture row for this sprint.
Post-run host health: external Gitea health passed, systemd-networkd remained active with the same 19:08:50Z restart timestamp, active_job_containers=0, airgap_containers=0, actions_networks=0, and host memory was stable.

Runs 3136, 3137, and 3142 Refresh

Latest checked bounded suite-shard CI runs: Gitea Actions runs 3136, 3137, and 3142, workflow test-manifest-execution.yml.
Dispatch for these runs: mode=manual, full_sweep=true, execute=true, runtime_cost=small, suite_shard_total=8.
Run 3136, small shard 2/8: 49 selected, 49 executed, 0 blocked, 0 failures, readiness ready.
Run 3137, small shard 3/8: 49 selected, 49 executed, 0 blocked, 1 failed, readiness ready because the suite is still report-only.
Failure group: tests-doctor-stellaops-doctor-plugin-storage-tests failed before test execution during restore/build because CI looked for /workspace/stella-ops.org/NuGet.config. Local property inspection showed root Directory.Build.props resolved StellaOpsRepoRoot one directory above the checkout.
Current classification: stale test placement plus repo-root build property bug, not a Doctor product behavior bug. The test project lived under the legacy top-level tests tree (the Doctor subtree), imported the root props, referenced stale Doctor APIs, and was missing module-owned project placement.
Implemented local fix: root Directory.Build.props resolves StellaOpsRepoRoot to $(MSBuildThisFileDirectory); the suite moved to src/Doctor/__Tests/StellaOps.Doctor.Plugin.Storage.Tests; project references now point at module-local Doctor plugin and shared Doctor library paths; stale DoctorStatus/DoctorPluginContext usage was updated; manifest and inventory rows now use doctor-stellaops-doctor-plugin-storage-tests.
Local validation: focused dotnet test src/Doctor/__Tests/StellaOps.Doctor.Plugin.Storage.Tests/StellaOps.Doctor.Plugin.Storage.Tests.csproj ... passed 7/7; dotnet msbuild -getProperty:RestoreConfigFile points at the checkout NuGet.config; manifest, trait, fixture, and CI-routing validators returned ok: true.
Pushed CI proof: run 3142 on commit 15fae9b1f0daee8ae7665dc9f3d756acde4c3747 reran small shard 3/8 and reported 49 selected, 49 executed, 0 blocked, 0 failures, readiness ready. This closes the Doctor storage restore failure as a stale placement/build-root issue.

Run 3146 / 3152 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3146, workflow test-manifest-execution.yml, commit 7bd1b7ea685514bc21ca31a6c52e5bd1a7bcf2de.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=small, suite_shard_total=8, suite_shard_index=3.
Evidence path: job log dashboard from small shard job 4131, plus refreshed history snapshot at TestResults/gitea-test-history-3146/.
Small shard result: 49 selected, 49 executed, 0 blocked, 2 failed, readiness ready because the failures were report-only.
Failure group 1: authority-stellaops-auth-serverintegration-tests failed TenantHeaderCallSiteConformanceTests.NoTenantHeaderUsageOutsideAllowList. New literals appeared in Findings fixture tests and in SbomService/Scanner worker tenant paths. Classification: Findings files were stale fixture cleanup; SbomService and Scanner worker were product-quality tenant handling bugs because they still accepted or forwarded legacy header-shaped tenant identity.
Failure group 2: tests-telemetry-stellaops-telemetry-core-tests failed restore with unversioned PackageReference items. Classification: stale duplicate top-level test project. The module-owned Telemetry core test project already exists under src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core.Tests.
Implemented local fixes: moved Telemetry P0ProductMetricsTests into the module-owned Telemetry test project and deleted the stale top-level project; refreshed manifest/routing inventories to 597 suites / 198 fixture roots / 23 CI-gap rows; removed new raw tenant-header literals from Findings fixtures; changed SbomLearnForwarder to resolve tenant through IStellaOpsTenantAccessor; changed Scanner worker tenant resolution to ignore legacy x-tenant-id metadata and prefer canonical scan metadata keys.
Local validation: Authority native xUnit conformance passed 1/1; SbomLearnForwarderTests passed 3/3 targeted and SbomService project passed 94/94; Scanner worker tenant classes passed 11/11 targeted and the full worker project passed 185/185; Telemetry core project passed 283/283; manifest, trait, fixture, and CI-routing validators returned ok: true.
Pushed CI proof: run 3152 on commit 0d040bb332a2623937c715a802932442bb897b38 reran bounded small shard 4/8 and reported 49 selected, 49 executed, 0 blocked, 0 failures, readiness ready. The authority-stellaops-auth-serverintegration-tests suite passed 43/43 in job 4143, and the stale tests-telemetry-stellaops-telemetry-core-tests failure row no longer appears in the dashboard/history snapshot at TestResults/gitea-test-history-3152/.

Run 3153 Refresh

Latest checked scheduled-nightly CI run: Gitea Actions run 3153, workflow test-manifest-execution.yml, commit 0d040bb332a2623937c715a802932442bb897b38.
Dispatch: schedule 17 1 * * *; large, medium, and small jobs ran under nightly full-sweep planning.
Large shard result: 42 selected, 3 executed, 3 blocked, 4 failure/blocker rows, readiness ready. The executed failure was web-stellaops-web; the remaining large rows were explicit manual-live or Docker/Testcontainers blockers.
Medium shard result: 164 selected, 5 executed, 0 blocked, 2 failures, readiness ready.
Failure group 1: web-stellaops-web failed in CI batch 11/52 because stale DashboardV3 specs rendered self-fetching lens children without providing their API dependencies. Local CI-shaped batch 11/52 now passes after adding the lens provider graph.
Failure group 2: tests-stellaops-airgapfederation-integrationtests-stellaops-airgapfederation-integrationtests failed before execution because the project referenced /workspace/stella-ops.org/git.stella-ops.org/nuget.config; the Linux checkout contains NuGet.config. Classification: case-sensitive restore-path bug in the top-level test project, not AirGapFederation product logic.
Failure group 3: root-stellaops-docs failed as npm error Missing script: "test". Classification: stale manifest command mapping / runner preflight gap. The root package is a tooling package without a test script, so the suite should block before launching npm unless a real root test command is added.
Closure proof: AirGapFederation now builds its RestoreConfigFile from the checkout root and exact NuGet.config filename; push run 3162 selected that suite, executed it, and passed 4/4. run-test-plan.py now verifies expected npm scripts for npm package rows; manual medium shard run 3166 reports root-stellaops-docs as blocked before npm with runner prerequisites missing: npm package package.json lacks scripts.test.

Run 3166 Refresh

Latest checked manual medium shard CI run: Gitea Actions run 3166, workflow test-manifest-execution.yml, commit d4c9f2c9b0f6aed645b6948786a25676f99d9422.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=medium, suite_shard_total=8, suite_shard_index=3.
Medium shard result: 21 selected, 15 executed, 6 blocked, 7 failure/blocker rows, readiness ready.
Closed runner-contract row: root-stellaops-docs blocked before execution because package.json lacks scripts.test; the generic npm error Missing script: "test" failure is gone.
Docker/Testcontainers blockers: 5 suites blocked before execution because Docker/Testcontainers is unavailable on the runner.
New executed failure: graph-stellaops-graph-api-tests failed 1/99. Local reproduction identified GraphUnsupportedRuntimeFeaturesIntegrationTests.Query_WithIncludeOverlays_ReturnsNotImplemented_InLiveRuntime failing with Npgsql.PostgresException 42P01: relation "graph.graph_nodes" does not exist; classification is Graph API test fixture migration setup, not a Graph query product assertion.
Local fix: GraphApiPostgresFixture now uses canonical schema graph and pre-runs Graph Indexer persistence startup migrations before any seed helper inserts into graph.graph_nodes. Local full project validation passed 99/99.
Observability finding: the CI dashboard only surfaced the aggregate MTP project failure and a noisy referenced-log tail, so the runner now decodes appended referenced failure logs to avoid UTF-16 null-character noise in dashboard failure details.

Run 3173 Refresh

Latest checked manual medium shard CI run: Gitea Actions run 3173, workflow test-manifest-execution.yml, commit 1efedc042bf2da11c23ec07c9538b6acf801b835.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=medium, suite_shard_total=8, suite_shard_index=3.
Medium shard result: 21 selected, 15 executed, 6 blocked, 7 failure/blocker rows, readiness ready.
Closed Graph row: graph-stellaops-graph-api-tests passed 99/99 in job 4188; the previous graph.graph_nodes missing-table failure did not recur.
New executed failure: smremote-stellaops-smremote-service-tests failed 1/27. The dashboard now names the suite, test method, assertion, and source location: EphemeralKeyHardeningTests.RotationService_EvictsExpiredEphemeralKey_AndEmitsRotatedEvent, Expected at least one rotation, got 0, /src/EphemeralKeyHardeningTests.cs:126.
Local classification: the SmRemote rotation test passes alone through the native xUnit runner but fails in the full project. The failure is stale/flaky test isolation around wall-clock TTL waiting plus process-static ephemeral audit/registry state under parallel test factories, not a confirmed product rotation bug.
Local fix: the test now resets both EphemeralKeyAuditLog and EphemeralKeyRegistry, injects a deterministic manual TimeProvider, and advances test time past the TTL before calling EphemeralKeyRotationService.Sweep(). Local full SmRemote project validation passed 27/27.

Run 3179 Refresh

Latest checked manual medium shard CI run: Gitea Actions run 3179, workflow test-manifest-execution.yml, commit 4598501dbca0ccbd40cae58758888d76c7ca24ef.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=medium, suite_shard_total=8, suite_shard_index=3.
Medium shard result: 21 selected, 15 executed, 6 blocked, 6 failure/blocker rows, readiness ready; job 4200 ran on stella-runner-1 from 2026-05-25T03:50:41Z to 2026-05-25T04:12:11Z.
Closed SmRemote row: smremote-stellaops-smremote-service-tests passed 27/27 in job 4200; the prior RotationService_EvictsExpiredEphemeralKey_AndEmitsRotatedEvent failure did not recur.
Persisting non-pass rows are explicit runner blockers only: five Docker/Testcontainers prerequisites and the root package.json missing scripts.test contract.
Historical dashboard validation: refreshed TestResults/gitea-test-history-3179/gitea-test-history.md records run/job IDs, runner names, timestamps, shard/count/readiness summaries, and failure detail job IDs.

Run 3167 Refresh

Latest checked Notify coverage CI failure: Gitea Actions run 3167, workflow e2e-notify-coverage.yml, scheduled full 4-phase run, commit d4c9f2c9b0f6aed645b6948786a25676f99d9422.
Failing job: 4175, Full 4-phase run (mock receivers), runner apparmor-host-runner.
Failure line: Docker could not start Mailpit because Bind for 0.0.0.0:1025 failed: port is already allocated.
Classification: runner/environment static host-port collision, not a Notify connector assertion failure. The job failed before the harness could prove channel delivery.
Fix direction: the mock-receiver compose no longer pins container/network/volume names or restartable services, the workflow now assigns a per-run compose project and per-run Mailpit SMTP/HTTP and echo host ports, and the harness accepts --mailpit-smtp-port so the Email connector uses the isolated SMTP host port.
Evidence state: local harness/build/YAML validations passed without host/server operations: the full StellaOps.E2E.NotifyChannelCoverage.Tests MTP project passed 118/118, the harness Release build succeeded with 0 warnings/errors, workflow/compose YAML parsed, and git diff --check passed. Commit a29604627c push checks also passed: docs CLI run 3189, architecture run 3190, and manifest execution run 3191. Run 3191 is no-overtrigger evidence only: it selected 43 suites, executed 2 report-only Go suites, skipped the Notify E2E row by push CI policy, and skipped all full-sweep jobs. Full scheduled proof remains pending because the lane does not run on ordinary push and should not be manually dispatched while host-safety review is active.

Run 3196 Refresh

Latest checked Workflow Renderer scheduled run: Gitea Actions run 3196, workflow workflow-renderer.yml, commit 267f0cb6b7f4aec05b712e873de5bbe48168c1f4.
Fast lane: job 4231 completed successfully on stella-runner-1.
Explicit slow lane: job 4232 was still in_progress on stella-runner-2 after the log reached DocumentProcessingWorkflowRenderBenchmark measured render 1/3.
Classification: CI lane contract/runtime budgeting issue, not a confirmed renderer product assertion failure. The job first runs repeated tools/elk-stress Best-effort benchmark coverage, then the NUnit explicit test pass repeats the heavy benchmark loop in Debug by default.
Fix direction: run the explicit NUnit pass in Release configuration and bound the NUnit DocumentProcessingWorkflowRenderBenchmark to one warmup plus one measured run. Keep repeated benchmark coverage in the preceding tools/elk-stress step where progress and summary are clearer. Because run 3196 later stayed active past the workflow’s nominal 90-minute timeout, also add shell-level budgets: 25 minutes for tools/elk-stress and 60 minutes for the NUnit explicit pass, with explicit error messages if either budget is exceeded.
Evidence state: local YAML validation and Release test-project validation passed without host/server operations: workflow YAML parsed, StellaOps.Workflow.Renderer.Tests.csproj built in Release with 0 warnings/errors, and Release test discovery completed without executing the explicit slow renders. Commit f3e379a74c push checks also passed: docs CLI run 3197, architecture run 3198, and manifest execution run 3199. Run 3199 is no-overtrigger evidence only: it selected 40 suites, executed 2 report-only Go suites, had 0 failures/0 blockers, readiness ready, and skipped all full-sweep jobs. Follow-up evidence commit 3a1378caf3 also passed docs CLI run 3200, architecture run 3201, and manifest run 3202; run 3202 selected/executed 0 suites, marked 597 not-selected, printed the expanded zero-suite dashboard, and skipped full sweeps. Scheduled run 3196 later completed successfully on the previous commit: fast lane job 4231 passed, and explicit slow-render job 4232 passed from 2026-05-25T06:00:29Z to 07:36:20Z. That is product-signal evidence but not final proof for the patched lane because it ran before the Release/shell-timeout workflow update and exceeded the intended 90-minute budget.

Run 3209 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3209, workflow test-manifest-execution.yml, commit 1bf4bc4014b4cbf46a218702053d0919a2be9668.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=small, suite_shard_total=8, suite_shard_index=4.
Empty shard jobs: large job 4258 and medium job 4259 completed successfully with 0 selected because the dispatch requested runtime_cost=small.
Small shard result: job 4260 ran on stella-runner-1 from 2026-05-25T07:45:28Z to 08:13:42Z; dashboard summary reported readiness ready, 48 selected suites, 48 executed suites, 0 blocked suites, and 0 failures.
Closed shard scope: small shard 5/8 is executed-pass evidence under the bounded manual full-sweep lane. Notable rows in this shard include releaseorchestrator-stellaops-agent-ssh-e2e-tests passing its executable path with 1 passed and 4 skipped, scanner-stellaops-scanner-integration-tests, scanner-stellaops-scanner-reachabilitydrift-tests, timeline-stellaops-timelineindexer-tests, and vexhub-stellaops-vexhub-core-tests.
Remaining scope: this run does not close the full all-suite mission. Small shards 6/8-8/8, non-small runtime shards, Docker/Testcontainers-capable execution, manual-live operator rows, scheduled Notify proof, and patched Workflow Renderer scheduled/controlled proof still require current CI evidence.

Run 3213 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3213, workflow test-manifest-execution.yml, commit abe55fc98140a2a17904eedaae90c9f7b3721d06.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=small, suite_shard_total=8, suite_shard_index=5.
Empty shard jobs: large job 4268 and medium job 4269 completed successfully with 0 selected because the dispatch requested runtime_cost=small.
Small shard result: job 4270 ran on stella-runner-1 from 2026-05-25T08:21:11Z to 08:49:40Z; dashboard summary reported readiness ready, 48 selected suites, 48 executed suites, 0 blocked suites, and 1 non-blocking failed suite.
Failure row: concelier-stellaops-concelier-connector-vndr-apple-tests failed 3/13. Dashboard detail surfaced Xunit.MicrosoftTestingPlatform.XunitException: System.InvalidOperationException : No canned response registered for GET https://support.example.com/en-us/HT214108.
Local classification: stale Linux fixture casing in the test harness. AppleConnectorTests.SeedDetail() registered fixture filenames ht214108.html and ht215500.html, but the committed fixture files are HT214108.html and HT215500.html. Windows passed because the file system is case-insensitive; Linux CI consumed the response once during retry/error handling and then reported the missing canned response.
Local fix: use exact fixture casing in AppleConnectorTests. Validation passed locally: dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests/StellaOps.Concelier.Connector.Vndr.Apple.Tests.csproj --disable-build-servers /m:1 /p:BuildInParallel=false /p:UseSharedCompilation=false /p:RestoreDisableParallel=true /nodeReuse:false --no-restore --logger "console;verbosity=minimal" passed 13/13, git ls-files --error-unmatch confirmed both uppercase fixture files, and rg -n "ht214108|ht215500" returned no lowercase literals.
Closure path: run 3213 stayed open until the same shard was rerun on the pushed fix.

Run 3217 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3217, workflow test-manifest-execution.yml, commit f4ce55a2c50a8718bafef50b1780682d64bd6d25.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=small, suite_shard_total=8, suite_shard_index=5.
Empty shard jobs: large job 4278 and medium job 4279 completed successfully with 0 selected because the dispatch requested runtime_cost=small.
Small shard result: job 4280 ran on stella-runner-2 from 2026-05-25T08:58:48Z to 09:27:04Z; dashboard summary reported readiness ready, 48 selected suites, 48 executed suites, 0 blocked suites, and 0 failures.
Closed shard scope: small shard 6/8 is executed-pass evidence under the bounded manual full-sweep lane. The previously failing concelier-stellaops-concelier-connector-vndr-apple-tests row passed 13/13 on Linux CI after the fixture-casing fix.
Remaining scope: small shards 7/8 and 8/8, non-small runtime shards, Docker/Testcontainers-capable execution, manual-live operator rows, scheduled Notify proof, and patched Workflow Renderer scheduled/controlled proof still require current CI evidence.

Run 3218 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3218, workflow test-manifest-execution.yml, commit f4ce55a2c50a8718bafef50b1780682d64bd6d25.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=small, suite_shard_total=8, suite_shard_index=6.
Empty shard jobs: large job 4282 and medium job 4283 completed successfully with 0 selected because the dispatch requested runtime_cost=small.
Small shard result: job 4284 ran on stella-runner-2 from 2026-05-25T09:28:13Z to 09:56:17Z; dashboard summary reported readiness ready, 48 selected suites, 48 executed suites, 0 blocked suites, and 0 failures.
Closed shard scope: small shard 7/8 is executed-pass evidence under the bounded manual full-sweep lane. Notable visibility checks include tests-stellaops-interop-tests reporting 11 passed and 38 skipped, and releaseorchestrator-stellaops-agent-winrm-e2e-tests reporting 1 passed and 3 skipped.
Remaining scope: small shard 8/8, non-small runtime shards, Docker/Testcontainers-capable execution, manual-live operator rows, scheduled Notify proof, and patched Workflow Renderer scheduled/controlled proof still require current CI evidence.

Run 3222 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3222, workflow test-manifest-execution.yml, commit 4a0c90adb2313673b54fb64dc36c5a4309ccd55a.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=small, suite_shard_total=8, suite_shard_index=7.
Empty shard jobs: large job 4292 and medium job 4293 completed successfully with 0 selected because the dispatch requested runtime_cost=small; targeted job 4291 skipped as expected.
Small shard result: job 4294 ran on stella-runner-1 from 2026-05-25T09:57:25Z to 10:23:47Z; dashboard summary reported readiness ready, 48 selected suites, 48 executed suites, 0 blocked suites, and 0 failures.
Closed shard scope: small shard 8/8 is executed-pass evidence under the bounded manual full-sweep lane. The final rows included Scanner, Signals, Tools, and VexLens suites, with vexlens-stellaops-vexlens-webservice-tests passing 7/7.
Remaining scope: all bounded small runtime shards now have current CI pass evidence. Non-small runtime shards, Docker/Testcontainers-capable execution, manual-live operator rows, scheduled Notify proof, and patched Workflow Renderer scheduled/controlled proof still require current CI evidence.

Run 3226 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3226, workflow test-manifest-execution.yml, commit 8beca4f7bb1635578d66b2029107469e975a7a94.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=medium, suite_shard_total=8, suite_shard_index=0.
Empty shard jobs: large job 4302 and small job 4304 completed successfully with 0 selected because the dispatch requested runtime_cost=medium; targeted job 4301 skipped as expected.
Medium shard result: job 4303 ran on stella-runner-2 from 2026-05-25T10:31:41Z to 10:47:39Z; dashboard summary reported readiness ready, 21 selected suites, 15 executed suites, 6 blocked suites, and 0 blocking failures.
Closed shard scope: executable rows in medium shard 1/8 are pass evidence. The passed rows include stellaops-signer-tests at 553 tests, stellaops-policy-determinization-tests at 438 tests, Scanner language/callgraph/storage suites, and workflow-stellaops-workflow-engine-tests at 185 tests.
Blocked scope: six rows remain runner-prerequisite blockers because Docker/Testcontainers is unavailable on the runner: tests-stellaops-integration-platform, binaryindex-stellaops-binaryindex-persistence-tests, concelier-stellaops-concelier-schemaevolution-tests, concelier-stellaops-excititor-persistence-tests, jobengine-stellaops-scheduler-webservice-tests, and releaseorchestrator-stellaops-releaseorchestrator-environment-tests.
Remaining scope: medium shards 2/8-8/8, all large shards, Docker/Testcontainers-capable execution, manual-live operator rows, scheduled Notify proof, and patched Workflow Renderer scheduled/controlled proof still require current CI evidence.

Run 3227 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3227, workflow test-manifest-execution.yml, commit 8beca4f7bb1635578d66b2029107469e975a7a94.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=medium, suite_shard_total=8, suite_shard_index=1.
Empty shard jobs: large job 4306 and small job 4308 completed successfully with 0 selected because the dispatch requested runtime_cost=medium; targeted job 4305 skipped as expected.
Medium shard result: job 4307 ran on stella-runner-2 from 2026-05-25T10:49:39Z to 11:16:20Z; dashboard summary reported readiness ready, 21 selected suites, 14 executed suites, 7 blocked suites, 2 failed executable suites, 9 total failures including blockers, and 0 blocking failures.
Fixed executable scope: concelier-stellaops-concelier-webservice-tests failed 1/329 in the Linux full project run. Local full native xUnit reproduction identified CanonicalAdvisoryEndpointTests.GetById_ReturnsOk_WhenCanonicalExists, not the dashboard-tail Federation hint, as the failing test: the canonical fixture mocked ICanonicalAdvisoryService but still used the real database-backed IInterestScoringService. After replacing that service with a mock, the native canonical class passed 16/16 and the native federation class passed 10/10. Pushed CI proof remains pending.
Fixed executable scope: findings-stellaops-findings-ledger-tests failed 7/221. Local full native run reproduced the failures, grouped them as stale production-surface expectations plus a shared Postgres fixture wiring gap in VulnExplorerEndpointsIntegrationTests, and after the test fixes StellaOps.Findings.Ledger.Tests.exe -xml TestResults/local-findings-ledger-full-after-fix2.xml passed 221/221 in 122.054s.
Blocked scope: seven rows remain runner-prerequisite blockers because Docker/Testcontainers is unavailable on the runner: tests-stellaops-integration-proofchain, attestor-stellaops-attestor-oci-tests, notify-stellaops-notify-persistence-tests, policy-stellaops-policy-gateway-tests, remediation-stellaops-remediation-tests, scanner-stellaops-scanner-worker-tests, and unknowns-stellaops-unknowns-persistence-tests.
Closure run: after the fixes were pushed, run 3235 on commit cb6dcb8ebbfc2ed1f1772f3d7f3f4e2c2f3e70a7 reran the same manual full-sweep medium shard 2/8. Job 4324 ran on stella-runner-1 from 2026-05-25T12:22:19Z to 12:48:12Z, selected 21 suites, executed 14, blocked the same 7 Docker/Testcontainers suites, reported readiness ready, and had 0 blocking failures. concelier-stellaops-concelier-webservice-tests passed 329/329 and findings-stellaops-findings-ledger-tests passed 221/221.
Remaining scope: continue medium shards 3/8-8/8, all large shards, Docker/Testcontainers-capable execution, manual-live operator rows, scheduled Notify proof, and patched Workflow Renderer scheduled/controlled proof.

Run 3239 Refresh

Latest checked bounded suite-shard CI run: Gitea Actions run 3239, workflow test-manifest-execution.yml, commit 850b7db52eafac30eb489bb93e8facbbf614d30ee.
Dispatch: mode=manual, full_sweep=true, execute=true, runtime_cost=medium, suite_shard_total=8, suite_shard_index=2.
Empty shard jobs: targeted job 4332 skipped; large job 4333 and small job 4335 completed successfully with 0 selected because the dispatch requested runtime_cost=medium.
Medium shard result: job 4334 ran on stella-runner-1 from 2026-05-25T12:56:33Z to 13:25:11Z; dashboard summary reported readiness ready, 21 selected suites, 14 executed suites, 7 Docker/Testcontainers blockers, 1 failed executable suite, and 0 blocking failures.
Fixed executable scope pending CI proof: cryptography-stellaops-cryptography-tests failed 3/279 in RandomBclCallSiteConformanceTests.NoBclRandomUsageOutsideAllowList, HmacBclCallSiteConformanceTests.NoBclHmacUsageOutsideAllowList, and AeadBclCallSiteConformanceTests.NoBclAeadUsageOutsideAllowList. The dashboard failure details named direct BCL crypto call sites in src/EvidenceLocker/__Libraries/StellaOps.EvidenceLocker.Export/Backup/LocalKmsBackupEncryptor.cs, src/EvidenceLocker/__Tests/StellaOps.EvidenceLocker.Export.Tests/EncryptedBundleExportTests.cs, and src/EvidenceLocker/__Tests/StellaOps.EvidenceLocker.Export.Tests/LocalKmsBackupEncryptorTests.cs.
Local fix: LocalKmsBackupEncryptor now routes randomness, AEAD, and HMAC through ICryptoRandom, IAeadAlgorithm, and IHmacAlgorithm; test fixture keys are deterministic arrays rather than BCL RNG output. The conformance tests now prefer git grep -l to identify tracked files containing forbidden tokens and retain the safe directory traversal fallback with bin/obj/node_modules/dist pruning, reparse-point skipping, and visited-directory guards.
Local validation: dotnet test src/EvidenceLocker/__Tests/StellaOps.EvidenceLocker.Export.Tests/StellaOps.EvidenceLocker.Export.Tests.csproj --no-build -v normal passed 97/97. Native xUnit conformance validation passed exactly 3/3 cases in 4.255s: StellaOps.Cryptography.Tests.exe -class StellaOps.Cryptography.Tests.RandomBclCallSiteConformanceTests -class StellaOps.Cryptography.Tests.HmacBclCallSiteConformanceTests -class StellaOps.Cryptography.Tests.AeadBclCallSiteConformanceTests.
Blocked scope: seven rows remain runner-prerequisite blockers because Docker/Testcontainers is unavailable on the runner: tests-stellaops-integration-reachability, authority-stellaops-authority-tests, notify-stellaops-notify-queue-tests, policy-stellaops-policy-persistence-tests, releaseorchestrator-stellaops-releaseorchestrator-evidencethread-tests, remediation-stellaops-remediation-webservice-tests, and unknowns-stellaops-unknowns-webservice-tests.
Remaining scope: re-run medium shard 3/8 after the fix is pushed; then continue medium shards 4/8-8/8, all large shards, Docker/Testcontainers-capable execution, manual-live operator rows, scheduled Notify proof, and patched Workflow Renderer scheduled/controlled proof.

Grouped Findings

Group	Suites	Classification	Production-quality direction
Frontend browser-test harness drift, CI termination, mixed command contract, and AOC BufferSource bug	`web-stellaops-web`	Stale Node/browser harness was fixed; run `3031` then exposed a real browser/runtime product bug in DSSE verification payload normalization	Keep the Node evidence guard in its Node/Vitest lane. Normalize all AOC WebCrypto inputs structurally and pass fresh `ArrayBuffer` instances to `verify`/`digest`. After the fix is pushed, re-run the manual large shard to prove Web passes. Longer term, split Web unit/browser/E2E manifest rows so manual-live E2E is not conflated with Angular unit batches.
DashboardV3 self-fetching lens provider drift	`web-stellaops-web` in scheduled run `3153`	Stale Web unit-test harness; local CI-shaped repro and fix complete, pushed CI proof pending	Dashboard tests that render `DashboardV3Component` must stub the full self-fetching lens API graph (`RELEASE_DASHBOARD_API`, `RELEASE_ENVIRONMENT_API`, `SECURITY_FINDINGS_API`, `ASSURANCE_PACKS_API`, `RELEASE_EVIDENCE_API`, `VEX_DECISIONS_API`, `SECURITY_OVERVIEW_API`, `RISK_API`, and active tenant session) so link/scope assertions test navigation behavior instead of failing on DI setup drift.
AirGapFederation case-sensitive NuGet config path	`tests-stellaops-airgapfederation-integrationtests-stellaops-airgapfederation-integrationtests` in scheduled run `3153`; push run `3162`	Closed as top-level .NET test project restore-path bug; pushed CI executed-pass proof complete	Keep restore config paths derived from the checkout root and exact `NuGet.config` casing. Use `dotnet msbuild -getProperty:RestoreConfigFile` as a focused guard for top-level test projects that import root props.
Root docs npm command mapping	`root-stellaops-docs` in scheduled run `3153`; manual medium run `3166`	Closed as stale manifest runner contract; CI now blocks before npm	Do not add fake root `npm test` scripts. Either define a real root test command or have manifest execution block npm package rows that lack their expected script before npm starts, with a dashboard reason that names the missing script and package path.
Graph API fixture migration gap	`graph-stellaops-graph-api-tests` in manual medium run `3166`; run `3173`	Closed as Graph API test harness bug; pushed CI executed-pass proof complete	Graph API integration fixtures must use canonical schema `graph` and pre-run Graph Indexer persistence migrations before seed helpers insert into `graph.graph_nodes`; dashboard logs must append referenced MTP failure logs cleanly so the exact failing test is visible.
SmRemote ephemeral-key rotation test isolation	`smremote-stellaops-smremote-service-tests` in manual medium run `3173`; run `3179`	Closed as stale/flaky test isolation; pushed CI executed-pass proof complete	TTL/rotation tests must use injected deterministic time instead of `Task.Delay`, and tests that use process-static ephemeral audit/registry state must reset it before assertions.
Manual live/operator environment missing	`tests-playwright-operator-trust-playwright`	Correctly manual-live; CI environment/seed blocker	Add an explicit live-stack preflight, health/auth diagnostics, and controlled seed requirements before this suite can be treated as passed or blocking.
.NET runner build/resource strategy	`signals-stellaops-signals-tests` after run `3006`; Docker/Testcontainers suites remain runner-contract blocked before build/test	CI tooling/runner strategy until full per-suite logs prove otherwise	Prebuild once per shard, run per-suite `dotnet test --no-build --no-restore`, and force low MSBuild/Roslyn parallelism (`/m:1`, `BuildInParallel=false`, `UseSharedCompilation=false`, node reuse off) or raise runner memory.
.NET 10 package compatibility warnings promoted to failures	Run `3000`: `cli-stellaops-cli-tests`, `policy-stellaops-policy-tests`; run `3006`: `signals-stellaops-signals-tests` via MSBuild package-resource path	Closed as a CI execution blocker by run `3031`; dependency hygiene remains	Prefer central package upgrades compatible with SDK `10.0.300`; until then keep `NETSDK1188` visible but non-fatal so suites can execute and expose real assertions.
.NET invariant globalization disables Unicode NFC normalization	Run `3098`: `tests-stellaops-testing-determinism-properties` medium shard	Runner/workflow configuration issue with production hardening implication	Do not weaken Unicode determinism tests. Stop setting `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1`, require full globalization/ICU for suites tagged `dotnet-full-globalization`, and block with an explicit runner prerequisite if invariant globalization is reintroduced.
Registry compatibility suite ran without Docker	Run `3127`: failed as dynamic-skip fixture output; run `3132`: `tests-stellaops-infrastructure-registry-testing-tests` blocks before execution on small shard 1/8	Closed as runner-contract / manifest infra misclassification; Docker-capable pass evidence still required	Keep Docker/Testcontainers/registry prerequisites in the canonical manifest source so Dockerless runners block before `dotnet test`. Add a Docker-capable runner before treating this suite as product pass/fail evidence.
Replayable verdict golden mismatch	Run `3127`: `tests-stellaops-e2e-replayableverdict` failed on small shard 1/8; run `3132`: row no longer appears in failure details	Closed as stale golden fixture data	Keep the updated bundle manifest and pinned verdict output at the deterministic current CGS hash, plus the guard that the output verdict file’s `cgsHash` matches manifest `expectedOutputs.verdictHash`.
Doctor storage suite restore failure	Run `3137`: `tests-doctor-stellaops-doctor-plugin-storage-tests` failed before execution looking for `/workspace/stella-ops.org/NuGet.config`; run `3142`: small shard `3/8` passed with 49/49 executed	Closed as stale test placement plus repo-root build property bug	Keep Doctor plugin storage tests under `src/Doctor/__Tests` and retain `dotnet msbuild -getProperty:RestoreConfigFile` as a focused guard when moving top-level test projects into module roots.
Authority tenant-header conformance failure	Run `3146`: `authority-stellaops-auth-serverintegration-tests` failed on new `X-Tenant-Id` literals in Findings fixtures plus SbomService/Scanner worker product paths; run `3152`: same suite passed 43/43	Closed as mixed stale fixture cleanup plus product tenant-resolution bugs	Keep the conformance test strict. Use canonical `X-StellaOps-TenantId`/claim-backed fixtures where possible; product forwarding must resolve tenant from `IStellaOpsTenantAccessor` or canonical scan metadata, never from legacy header-shaped keys.
Telemetry duplicate top-level project	Run `3146`: `tests-telemetry-stellaops-telemetry-core-tests` failed restore because stale top-level project had unversioned packages; run `3152`: stale failure row disappeared after consolidation	Closed as stale duplicate test placement	Keep Telemetry tests under `src/Telemetry/StellaOps.Telemetry.Core/StellaOps.Telemetry.Core.Tests`; do not retain top-level duplicate projects that bypass central package/version conventions.
Scanner worker worktree-root helper	Local full Scanner worker project run exposed `SurfaceManifestStageExecutorTests.ResolveRepositoryRoot()` failing when `.git` is a worktree file	Stale test helper found during adjacent validation	Repository-root helpers should accept `.git` files and/or `global.json` plus `src` so CI worktrees and local worktrees behave the same.
Notify e2e static host-port collision	Run `3167`: scheduled `e2e-notify-coverage.yml` full-run failed before harness execution because Mailpit could not bind host `0.0.0.0:1025`	Runner/environment collision from globally pinned compose resources	Keep local defaults for developer use, but make CI use per-run compose project names and dynamic host ports. Pass the selected Mailpit SMTP port into both the direct Mailpit transport and Email connector-backed delivery path.
Workflow Renderer explicit slow-lane overrun	Run `3196`: scheduled `workflow-renderer.yml` fast lane passed, explicit lane stayed active after `DocumentProcessingWorkflowRenderBenchmark` measured run `1/3`	CI lane contract/runtime budgeting issue	Run explicit NUnit tests in Release and keep the NUnit benchmark to one smoke sample; rely on `tools/elk-stress` for repeated benchmark iterations.
Apple connector fixture-case drift	Run `3213`: `concelier-stellaops-concelier-connector-vndr-apple-tests` failed 3/13 on Linux CI; run `3217`: same row passed 13/13 in bounded small shard `6/8`	Closed as stale Linux-only test fixture casing	Keep fixture filename references exact-case. Windows local validation is not enough for filename casing; Linux CI shard proof is required before closing future fixture-case failures.
Findings Ledger VEX integration fixture/wire drift	Run `3227`: `findings-stellaops-findings-ledger-tests` failed 7/221; run `3235`: same row passed 221/221	Closed as stale tests plus test-fixture wiring bug	VEX endpoint integration tests that share `FindingsLedgerPostgresFixture` must construct the web factory with that fixture’s connection string. Assertions should match the Console wire contract (`SCREAMING_SNAKE_CASE`) and the current production split where VEX decisions are stateful while unsupported fix-verification write paths remain truthful.
Concelier WebService full-suite stale fixture and failure-detail gap	Run `3227`: `concelier-stellaops-concelier-webservice-tests` failed 1/329; run `3235`: same row passed 329/329	Closed as stale test fixture/service isolation bug; dashboard parser hardening merged	Endpoint contract tests must replace optional enrichment services that are outside the test scope, especially database-backed services. CI dashboards must parse TRX failed-test details so outsiders see the actual failed test, message, and stack instead of a misleading output-tail hint.
Cryptography BCL call-site conformance regression	Run `3239`: `cryptography-stellaops-cryptography-tests` failed 3/279 in RNG/HMAC/AEAD call-site conformance; local fix complete, pushed CI proof pending	Real downstream production-quality regression plus conformance-test performance gap	Do not grow allow-lists for new production code. EvidenceLocker backup encryption must route through `ICryptoRandom`, `IAeadAlgorithm`, and `IHmacAlgorithm`; tests should use deterministic fixture keys. Repo-wide conformance scans should use tracked-file grep preselection and native xUnit runner targeting when MTP ignores `dotnet test --filter`.
Benchmark suites executed as tests	`binaryindex-stellaops-binaryindex-benchmarks`	Bad manifest command mapping / stale suite contract	Do not run BenchmarkDotNet projects via `dotnet test`. Add a CI smoke/unit project or run benchmarks through `dotnet run -c Release` only in weekly/manual benchmark lanes with explicit budgets.
Docker/Testcontainers integration requirements	9 Docker/Testcontainers suites listed below, including Concelier, Router, Scanner, Timeline, chaos, and parity rows	CI environment and manifest lane placement	Add runner labels/preflight for Docker/Testcontainers/Postgres and keep these in weekly/manual integration lanes until the runner contract is proven.
Scanner WebApplicationFactory/content-root failure	`scanner-stellaops-scanner-webservice-tests`	Run `2938` suggested a config/content-root issue; run `3000` showed the row also has hidden Docker/Testcontainers/Postgres prerequisites	Declare the hidden infra so Dockerless runners block before execution, then re-run on a Docker-capable runner before deciding whether any WebApplicationFactory fixture issue remains.
Scanner reachability unclear preview	`scanner-stellaops-scanner-reachability-tests`	Closed by run `3006`; the suite passed in pushed CI	No follow-up for this sprint unless a future run regresses.

Suite Details

`web-stellaops-web`

Manifest path: src/Web/StellaOps.Web/package.json.
CI policy: manual=required, gating=manual-live, currentState=report-only, runtime large.
Layers: browser E2E, E2E, frontend unit.
Seed/fixture roots: src/test-setup.ts, tests/e2e, e2e, output/playwright.
CI preview: command reached npm run test:ci.
Agent finding: the manifest currently represents a broad Web suite, while the runner maps package suites to npm run test:ci, which executes Angular unit batches rather than the separate Playwright E2E paths.
Local targeted reproduction: npm run test -- --batch-from=1 --batch-to=1 in src/Web/StellaOps.Web.
Local result: Angular bundle generation failed before executing specs because src/tests/evidence/no-bundle-mutation.guard.spec.ts imports node:fs, node:path, and uses __dirname, while tsconfig.spec.json only includes vitest/globals types and the Angular browser test builder compiles it.
Classification: stale harness/config and suite-contract ambiguity. The evidence guard is useful, but it belongs in a Node-oriented lane or needs explicit Node typing and isolation from browser compilation. Web unit, browser E2E, and live E2E should be separate manifest rows or explicit commands.
Fix started: the guard is excluded from Angular browser test compilation and npm run test:ci now runs it through vitest --environment node after the browser batches. Local validation passed for the standalone guard and the previously failing first Angular batch.
Follow-up evidence fix: run 3006 still reduced the live failure detail to Batch 1/35 failed.. The Web unit batch runner now prints the failed batch’s spec file paths and the exact npm run test -- --batch-from N --batch-to N rerun command so future CI logs identify the failure scope even when artifacts are unavailable.
Fresh local repro after run 3006: a side-agent ran npm ci --prefer-offline --no-audit --no-fund and npm run test -- --batch-from 1 --batch-to 1 on commit 8aa849f181; batch 1 passed locally with 12 files and 98 tests, and no tracked files changed.
Run 3031 result: the suite advanced past the old guard/import-key failures and produced a concrete failure tail. src/app/core/aoc/provenance-builder.spec.ts failed only verifies DSSE signatures using pre-auth encoding; Chrome Headless reported TypeError: Failed to execute 'verify' on 'SubtleCrypto': 4th argument is not instance of ArrayBuffer, Buffer, TypedArray, or DataView.
Current classification: confirmed Web AOC product/runtime bug around cross-realm BufferSource normalization. The fix uses structural ArrayBuffer.isView handling and fresh ArrayBuffer copies in signature-verifier.ts and checksum.util.ts.
Local validation after the fix: npm run test -- --batch-from 1 --batch-to 1 passed 12 files / 98 tests, and npm run test:evidence:bundle-guard passed 1/1. Pushed CI proof is still required.
Run 3038 early result: after the AOC BufferSource fix was pushed, the first Web Angular batch was killed by the CI runner before assertion output: Angular test process terminated by signal SIGKILL.
Current additional classification: CI runner resource pressure for the broad Angular unit batch. The batch runner now defaults to 6 files per batch and --max-old-space-size=2048 in CI, while local developer runs keep 12 files and 3072 MB. Operators can override with STELLAOPS_WEB_TEST_BATCH_SIZE and STELLAOPS_WEB_TEST_NODE_MEMORY_MB.
Local validation after the runner-resource fix: CI=true node scripts/run-unit-test-batches.mjs --print-batch 1 listed 6 files, and CI=true npm run test -- --batch-from 1 --batch-to 1 passed 6 files / 43 tests.

`tests-playwright-operator-trust-playwright`

Manifest path: src/__Tests/playwright/package.json (relocated 2026-05-30 from tests/playwright/; the date-stamped investigation paths below were captured pre-move and are left as historical record).
CI policy: manual=required, gating=manual-live, currentState=no-active-ci, runtime large.
Layers: browser E2E, E2E, operator-live.
Seed data: operator-baseline, operator-trust, output/evidence.
CI preview: failure at tests/playwright/_shared/auth.fixture.ts:94, inside the authenticated adminContext fixture.
Local code finding: the fixture loads src/Web/StellaOps.Web/scripts/live-frontdoor-auth.mjs, defaults to https://stella-ops.local, and requires live username/password/token context from environment.
Agent finding: npm --prefix tests/playwright run test:list listed the Playwright specs successfully, so package setup and test discovery are not the blocker.
Classification: missing controlled live stack and auth seed for CI, not a product bug until the stack is present and the flow still fails.
Required fix shape: a preflight that prints base URL, health/auth endpoint reachability, credential source presence, tenant, browser binary path, and where evidence was written. The suite should remain manual-live until that preflight is green on a designated runner.
Fix started: the manifest runner now blocks npm/playwright suites that require a live Stella Ops stack before launching Playwright when STELLA_BASE_URL or credential environment variables are missing. Local validation produced a blocked dashboard row and evidence log for the operator suite.

`.NET runner build/resource strategy`

Affected suites:
- cli-stellaops-cli-tests
- concelier-stellaops-concelier-federation-tests
- scanner-stellaops-scanner-reachability-tests
- scanner-stellaops-scanner-storage-oci-tests
- signals-stellaops-signals-tests
- possibly scanner-stellaops-scanner-webservice-tests
Agent finding: runner commands are plain per-suite dotnet test <csproj> invocations from the repo root. They do not currently prebuild once, use --no-build, disable restore for each suite, or constrain MSBuild/Roslyn parallelism.
Runner context: CI large shard was running on limited self-hosted resources; local targeted probes also hit MSBuild/Roslyn memory pressure before reaching test assertions.
Classification: CI tooling/runner strategy until full logs expose actual test failures.
Required fix shape:
- Generate a per-shard .NET prebuild phase for selected .csproj suites.
- Execute each suite with dotnet test --no-build --no-restore.
- Add conservative MSBuild properties: /m:1, /p:BuildInParallel=false, /p:UseSharedCompilation=false, /nodeReuse:false.
- Keep full per-suite logs and larger failure tails so a second run can distinguish real product failures from build/resource failures.

`.NET 10 NETSDK1188` failures

Affected suites from run 3000:
- cli-stellaops-cli-tests
- policy-stellaops-policy-tests
Historical run 2938 also showed related SDK/package warning failures in Concelier, Scanner Storage OCI, and Signals.
CI previews show NETSDK1188 warnings from packages such as Microsoft.CodeAnalysis.*, Microsoft.Testing.Platform, and Microsoft.Testing.Platform.MSBuild.
Repository context: src/Directory.Build.props globally treats warnings as errors, but xUnit v3-style *.Tests projects normally set TreatWarningsAsErrors=false. Projects that are not named *.Tests, benchmark projects, or projects with local overrides can still fail on SDK compatibility warnings.
Classification: dependency/toolchain hygiene. Do not classify these as feature regressions until the projects build under the intended SDK.
Implemented local fix: src/Directory.Build.props includes NETSDK1188 in WarningsNotAsErrors, keeping the warning in logs while preventing upstream package metadata from blocking execution.
Local validation:
- dotnet test src/Cli/__Tests/StellaOps.Cli.Tests/StellaOps.Cli.Tests.csproj ... passed after stale test repairs: 1513 total, 1507 passed, 6 skipped, 0 failed.
- dotnet test src/Policy/__Tests/StellaOps.Policy.Tests/StellaOps.Policy.Tests.csproj ... passed: 798 passed, 0 failed.
Pushed CI status: run 3006 proved cli-stellaops-cli-tests and policy-stellaops-policy-tests advance past NETSDK1188 and pass.
Follow-up fix: src/Directory.Build.props also sets MSBuildWarningsAsMessages=NETSDK1188; local Signals validation passed with 1448 tests and 0 failures.
Final pushed CI status: run 3031 proved signals-stellaops-signals-tests advances past the MSBuild-side NETSDK1188 failure and passes in CI. Central dependency upgrades remain the preferred long-term cleanup after license review.

`.NET full globalization / Unicode NFC`

Affected suite from run 3098: tests-stellaops-testing-determinism-properties.
CI preview: Expected string to be the same string, but they differ at index 0:.
Reproduction: setting DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 locally reproduces the Hangul/NFC behavior in UnicodeNormalizationDeterminismProperties.HangulJamoCombinationsNormalizeConsistently; running with DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=0 passes the two Unicode equivalence tests.
Code finding: CanonJson.Writer normalizes property names and string values to NormalizationForm.FormC, so invariant globalization is not a valid runtime condition for canonical JSON hash/evidence determinism.
Classification: runner/workflow configuration, not stale test. The existing tests are valuable because they catch a real precondition for deterministic canonical JSON.
Implemented fix: active Gitea .NET workflows now set DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=0, the determinism property suite declares dotnet-full-globalization, and run-test-plan.py blocks that suite with a clear runner prerequisite if invariant globalization is enabled.
Local validation: DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 one-suite manifest execution now reports status=blocked with the explicit prerequisite message; DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=0 dotnet test ... --filter "HangulJamoCombinationsNormalizeConsistently|KnownUnicodeEquivalentsProduceSameHashAfterNfc" passed 2/2.
Pushed CI proof is still required.

CLI stale tests found during adjacent validation

StellaOps.Cli.Tests.Integration.BinaryDiffIntegrationTests.ComputeDiffAsync_WithElfFixtures_ProducesModifiedFinding failed locally because FindRepositoryRoot() required a .git directory. Git worktrees use a .git file, so the test harness could not locate fixtures from a clean worktree.
StellaOps.Cli.Tests.Commands.MigrationModuleConsolidationTests.Build_ForScanner_ProducesSingleConsolidatedMigration failed locally because the Scanner source migration count was stale: expected 42, actual 45.
Classification: stale tests, not product bugs.
Implemented local fix: locate the repository root using global.json plus src and docs, and update the Scanner consolidation assertion to 45 with the current migrations 031_sbom_sources_webhook_secret_ref.sql, 032_sbom_sources_enum_and_run_shape.sql, and 033_scan_evidence_projections.sql.

`binaryindex-stellaops-binaryindex-benchmarks`

Manifest path: src/BinaryIndex/__Tests/StellaOps.BinaryIndex.Benchmarks/StellaOps.BinaryIndex.Benchmarks.csproj.
CI preview: NETSDK1188 warning from Microsoft.CodeAnalysis.CSharp.
Agent finding: the manifest marks it as category=Benchmark, weekly candidate/manual allowed, and zero estimated tests. The project contains BenchmarkDotNet [Benchmark] classes, has no facts/theories/assertions, and has no executable Program.cs entry point even though source comments indicate benchmark execution through dotnet run -c Release --filter ....
Classification: stale suite contract / bad command mapping.
Required fix shape: keep benchmarks in weekly/manual extended lanes, add a real benchmark executable runner with explicit timeout/resource budget, or create a separate small smoke test project for per-commit correctness.
Fix started: the manifest runner now blocks pure BenchmarkDotNet rows before execution and reports the benchmark-lane runner contract as status=blocked. Local validation with the real BinaryIndex manifest row produced blockedSuiteCount=1, failureCount=1, and a stable suite log.

Docker/Testcontainers suites

Affected suites:
- tests-stellaops-chaos-controlplane-tests
- tests-stellaops-chaos-router-tests
- tests-stellaops-parity-tests
- concelier-stellaops-concelier-federation-tests
- concelier-stellaops-concelier-persistence-tests
- router-stellaops-messaging-transport-valkey-tests
- scanner-stellaops-scanner-storage-oci-tests
- parts of scanner-stellaops-scanner-webservice-tests
- timeline-stellaops-timeline-webservice-tests
Manifest infra declares Docker/Testcontainers/Postgres for Concelier federation and Scanner Storage OCI.
Code finding: Scanner WebService has a shared Testcontainers Postgres path and explicit skip handling when Docker/Testcontainers cannot start, but build-time SDK/package failures happen before those skips can help.
Classification: CI runner contract. These suites need an explicit runner label/preflight and cached/offline container images before pass/fail can be trusted as product evidence.
Fix started: the manifest runner now checks Docker/Testcontainers prerequisites before launching suites with docker-testcontainers or testcontainers infra and reports missing Docker as status=blocked. Local validation with Docker removed from PATH produced blockedSuiteCount=1, failureCount=1, and a stable suite log before any dotnet test process launched.

`scanner-stellaops-scanner-webservice-tests`

Manifest path: src/Scanner/__Tests/StellaOps.Scanner.WebService.Tests/StellaOps.Scanner.WebService.Tests.csproj.
CI preview: stack begins at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder).
Code finding: Program.cs loads scanner defaults with options.BasePath = builder.Environment.ContentRootPath and adds ../etc/scanner.yaml; tests use ScannerApplicationFactory with in-memory overrides and many UseContentRoot(...) call sites.
Run 3000 preview did not expose enough stack detail to confirm a product or fixture bug, and the manifest row was missing hidden Testcontainers/Postgres requirements.
Implemented local fix: manifest refresh now sets this suite’s runtime infra to docker-testcontainers, postgres, and testcontainers instead of the misleading package-derived benchmarkdotnet signal. A local Dockerless one-suite plan now blocks before dotnet test with a stable runner-prerequisite log.
Classification: runner/manifest contract first. After a Docker-capable runner proves the prerequisites, any remaining WebApplicationFactory/content-root failure should be fixed at the fixture level.

`scanner-stellaops-scanner-reachability-tests`

Manifest path: src/Scanner/__Tests/StellaOps.Scanner.Reachability.Tests/StellaOps.Scanner.Reachability.Tests.csproj.
CI preview only shows a build line for StellaOps.DependencyInjection, not the actual failure.
Run 3006 result: passed in pushed CI after the runner and warning-policy fixes.
Classification: closed for this sprint.

CI Follow-up Matrix

Lane	Required condition	Current state
Per commit / PR	Targeted suites only; no broad rerun for docs/demo-only changes	Implemented by manifest planning; run `3030` proved a Web source change selected only `web-stellaops-web` and skipped execution by policy instead of triggering a full rerun.
Merge to main	Targeted main plan with dashboard warning when zero suites execute	Proven for zero-suite and policy-skipped push runs; keep observing new pushes after fixes.
Nightly	Execute non-live medium/small/large suites that are nightly candidates and publish dashboard	Not yet proven green.
Weekly / extended	Benchmarks, Docker/Testcontainers, and slow integration suites	Benchmark and Docker/Testcontainers runner-contract blockers are now explicit in run `3006`; pass evidence still requires the correct benchmark/Docker-capable runner.
Manual live	Operator Playwright and production-like live E2E with SCM/repository seeds	Blocked until controlled live stack, credentials, browser, and seed preflight are present.

Immediate Fix Backlog

Continue the bounded all-suite matrix with small shards 4-8 followed by medium and large shards, using status polling and completed-log collection only.
Keep observing the current Web manual-large proof; if Web remains too quiet during npm run test:ci, rely on the new selective live-output stream in the next run.
Add a controlled live operator environment before treating Playwright manual-live suites as pass/fail product evidence.
Add a benchmark executable lane or xUnit smoke coverage for pure BenchmarkDotNet projects.
Provide a Docker/Testcontainers-capable runner contract with cached/offline images before treating Docker-backed suite results as product evidence.
Re-run scanner-stellaops-scanner-webservice-tests on a Docker-capable runner after the manifest infra fix; only then reproduce and fix any remaining config/content-root stack.