Automated Test-Suite Overview

Audience: contributors extending test coverage or diagnosing CI failures.

This document is the conceptual map of the Stella Ops test strategy — the layers, categories, and quality gates that run from unit level up to chaos experiments. It explains how the pipeline is organized and why.

Target map, not a live registry. This overview describes the intended shape of the pipeline. For the authoritative, source-derived inventory of runnable suites and their current CI posture, use TEST_SUITE_CONSOLIDATION_DEEP_DIVE.md, TEST_PROJECT_CI_INVENTORY.md, and the trait taxonomy in TEST_TRAIT_TAXONOMY.md. The canonical tests live under src/__Tests/** and each module’s __Tests/ directory.

Build parameters – values such as {{ dotnet }} (runtime) and {{ angular }} (UI framework) are injected at build time.

Test Philosophy

The existing python-runtime-compat-verify.yml job also runs the Evidence volume archive controls in devops/compose/scripts/test_backup_evidence.py and the volume-backup.py --self-test detector controls. They invoke the real backup and verification entry points with a Docker-boundary stand-in and an unreachable Docker endpoint. Fixture archive/restore integrity and encryption delegation are source checks; they do not establish real cryptographic or installation recovery.

The workspace distribution guard runs on service publish contexts, Scanner bundles and the final CLI publication payload. Its self-test exercises nested archives, dependency metadata and embedded setup-interop markers. Artifact absence and notice delivery remain separate from semantic M1 tests.

The architecture pack includes the P13 retention-declaration inventory guard (P13RetentionClassConformanceTests). Its normal source inventory/growth gate is distinct from the strict pending-repair audit and from behavioral/live acceptance. The P13 inventory guide defines those limits and commands; the existing architecture workflow watches all source C#/SQL plus the detector’s policy inputs.

Core Principles

The architecture workflow also runs the curated OpenAPI currency guard. Its composer self-test and non-mutating check keep the committed static aggregate aligned with the owner inputs, including draft markers; runtime endpoint conformance remains a separate check.

1. Determinism as Contract: Scan verdicts must be reproducible. Same inputs → byte-identical outputs.
2. Offline by Default: Every test (except explicitly tagged “online”) runs without network access.
3. Evidence-First Validation: Assertions verify the complete evidence chain, not just pass/fail.
4. Interop is Required: Compatibility with ecosystem tools (Syft, Grype, Trivy, cosign) blocks releases.
5. Coverage by Risk: Prioritize testing high-risk paths over line coverage metrics.

The architecture workflow runs the Compose source binding census against the evaluated source graph. Exact owner worklists retain unresolved key, host and health contracts; source disposition does not imply runtime consumption.

Test Boundaries

The architecture workflow runs the test-truth source census. Its shrink-only worklist keeps environment-isolation and migration-input candidates bound to open owner tasks. Repeated native suite execution is separate acceptance evidence.

• Lattice/policy merge algorithms run in scanner.webservice
• Concelier/Excitors preserve prune source (no conflict resolution)
• Tests enforce these boundaries explicitly

Model taxonomy

See testing-strategy-models.mdfor the required test types per project model and the module-to-model mapping.

Layer Map

Test Categories (xUnit Traits)

[Trait("Category", "Unit")]           // Fast, isolated unit tests
[Trait("Category", "Property")]       // Property-based checks (sub-trait)
[Trait("Category", "Snapshot")]       // Golden/snapshot assertions (sub-trait)
[Trait("Category", "Integration")]    // Tests requiring infrastructure
[Trait("Category", "Contract")]       // Schema and API contract checks
[Trait("Category", "E2E")]            // Full end-to-end workflows
[Trait("Category", "AirGap")]         // Must work without network
[Trait("Category", "Interop")]        // Third-party tool compatibility
[Trait("Category", "Performance")]    // Performance benchmarks
[Trait("Category", "Chaos")]          // Failure injection tests
[Trait("Category", "Security")]       // Security-focused tests
[Trait("Category", "Live")]           // Opt-in upstream connector tests

Quality Gates

The architecture lane enforces the consumer census worklist. Its self-tests exercise real registration removal, stable source identities and missing owner rows. The source gate retains exact verification/withdrawal candidates; it does not assert that a baselined candidate has no runtime consumer.

The architecture lane checks the reviewed client route worklist. Its exact outcomes and authority checks guard regressions; full served-route closure remains separate, owned by the filed owner tasks while source, destination or owner composition is unverified.

The architecture lane enforces reviewed documentation currency with semantic self-tests and an exact shrink-only baseline. This focused gate does not close the full program census, whose unresolved route classifications remain explicit.

The architecture lane also validates the push-lane liveness detector’s hermetic controls and workflow inventory. The separate live history probe distinguishes completed failures from cancelled or missing runs; see the gate contract.

Architecture conformance prerequisites and failure evidence

The architecture pack includes PowerShell preflight checks and Docker Compose configuration-render checks. It requires pwsh and the Docker CLI with the Compose plugin, but does not require a Docker daemon, a database, or a running estate. The CI workflow provisions missing CLI prerequisites before the pack; it does not skip the checks when tools are absent.

On a failed full-pack run, tools/scripts/test/emit-mtp-failure-log.py decodes the MTP UTF-8/UTF-16 result logs and emits every failed case and stack trace to the job console. The workflow returns the original dotnet test exit code, independent of log emission or artifact-download availability. Re-verify the decoder with python3 tools/scripts/test/emit-mtp-failure-log.py --self-test.

Local Runner

# minimal run: unit + property + frontend tests
./scripts/dev-test.sh

# full stack incl. Playwright and lighthouse
./scripts/dev-test.sh --full

# category-specific
dotnet test --filter "Category=Unit"
dotnet test --filter "Category=AirGap"
dotnet test --filter "Category=Interop"

The script spins up PostgreSQL/Valkey via Testcontainers and requires:

• Docker ≥ 25
• Node 20 (for the Angular/Vitest unit lane and Playwright)

PostgreSQL Testcontainers

Multiple suites (Concelier connectors, Excititor worker/WebService, Scheduler) use Testcontainers with PostgreSQL for integration tests. If you don’t have Docker available, tests can also run against a local PostgreSQL instance listening on 127.0.0.1:5432.

Local PostgreSQL Helper

Some suites (Concelier WebService/Core, Exporter JSON) need a full PostgreSQL instance when you want to debug or inspect data with psql. A helper script is available under tools/postgres/local-postgres.sh:

# start a local PostgreSQL instance
tools/postgres/local-postgres.sh start

# stop / clean
tools/postgres/local-postgres.sh stop
tools/postgres/local-postgres.sh clean

By default the script uses Docker to run PostgreSQL 16, binds to 127.0.0.1:5432, and creates a database called stellaops. The connection string is printed on start and you can export it before running dotnet test if a suite supports overriding its connection string.

New Test Infrastructure (Epic 5100)

Run Manifest & Replay

Every scan captures a Run Manifest containing all inputs (artifact digests, feed versions, policy versions, PRNG seed). This enables deterministic replay:

# Replay a scan from manifest
stella replay --manifest run-manifest.json --output verdict.json

# Verify determinism
stella replay verify --manifest run-manifest.json

Evidence Index

The Evidence Index links verdicts to their supporting evidence chain:

• Verdict → SBOM digests → Attestation IDs → Tool versions

Golden Corpus

Located at src/__Tests/__Datasets/golden-corpus/, contains 50+ test cases:

• Severity levels (Critical, High, Medium, Low)
• VEX scenarios (Not Affected, Affected, Conflicting)
• Reachability cases (Reachable, Not Reachable, Inconclusive)
• Unknowns scenarios
• Scale tests (200 to 50k+ packages)
• Multi-distro (Alpine, Debian, RHEL, SUSE, Ubuntu)
• Interop fixtures (Syft-generated, Trivy-generated)
• Negative cases (malformed inputs)

Offline Testing

Inherit from NetworkIsolatedTestBase for air-gap compliance:

[Trait("Category", "AirGap")]
public class OfflineTests : NetworkIsolatedTestBase
{
    [Fact]
    public async Task Test_WorksOffline()
    {
        // Test implementation
        AssertNoNetworkCalls();  // Fails if network accessed
    }
}

Concelier OSV↔GHSA Parity Fixtures

The Concelier connector suite includes a regression test (OsvGhsaParityRegressionTests) that checks a curated set of GHSA identifiers against OSV responses. The fixture snapshots live in src/Concelier/__Tests/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/ and are kept deterministic so the parity report remains reproducible.

To refresh the fixtures when GHSA/OSV payloads change:

1. Ensure outbound HTTPS access to https://api.osv.dev and https://api.github.com.
2. Run UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj.
3. Commit the regenerated osv-ghsa.*.json files that the test emits (raw snapshots and canonical advisories).

The regen flow logs [Parity] messages and normalises recordedAt timestamps so the fixtures stay stable across machines.

CI Job Layout

flowchart LR
  subgraph fast-path
    U[xUnit] --> P[FsCheck] --> I1[Testcontainer API]
  end

  I1 --> FE[Jest]
  FE --> E2E[Playwright]
  E2E --> Lighthouse

  subgraph release-gates
    REPLAY[Replay Verify]
    INTEROP[Interop E2E]
    OFFLINE[Offline E2E]
    BUDGET[Unknowns Gate]
  end

  Lighthouse --> INTEG2[Concelier]
  INTEG2 --> LOAD[k6]
  LOAD --> CHAOS[Chaos Suite]
  CHAOS --> RELEASE[Attestation diff]

  RELEASE --> release-gates

Adding a New Test Layer

1. Extend scripts/dev-test.sh so local contributors get the layer by default.
2. Add a dedicated workflow in .gitea/workflows/.
3. Register the layer here (this document) and add its row to the test manifest (TEST_MANIFEST.yml); if it introduces a new quality gate, record the gate metric alongside the others in the Quality Gates table.
4. If the test requires network isolation, inherit from NetworkIsolatedTestBase.
5. If the test uses golden corpus, add cases to src/__Tests/__Datasets/golden-corpus/.

Related Documentation

• Testing README — complete testing documentation index
• Testing Strategy Models — required test types per project model
• Test Trait Taxonomy — canonical Category values and CI routing
• Test Suite Consolidation Deep Dive — source-derived suite inventory
• CI Lane Filters — how CI selects and gates by trait
• CI/CD Test Strategy — CI/CD integration details
• Offline Operation Guide
• Module Architecture Dossiers

Last updated 2026-05-31

Exact client owner worklist

The existing architecture lane runs check-client-route-owner-worklist.py --self-test and --filed. --filed verifies the filing manifest against its pinned filing baseline. It then checks current state against the current baseline and an actual full client census. Every open UNVERIFIED outcome must keep its original review metadata and a real owner task. Every raw unknown must be owner-filed or a filed finding. No task may close or be archived while one of its identities stays open, unless an independently validated resolution covers it. Settled identities release their row; the full child returns UNVERIFIED exit 2 while unknowns remain. A passing gate proves ownership integrity, not served-route closure. Filing receipt.