Automated Test-Suite Overview

Audience: contributors extending test coverage or diagnosing CI failures.

This document is the conceptual map of the Stella Ops test strategy — the layers, categories, and quality gates that run from unit level up to chaos experiments. It explains how the pipeline is organized and why.

Target map, not a live registry. This overview describes the intended shape of the pipeline. For the authoritative, source-derived inventory of runnable suites and their current CI posture, use TEST_SUITE_CONSOLIDATION_DEEP_DIVE.md, TEST_PROJECT_CI_INVENTORY.md, and the trait taxonomy in TEST_TRAIT_TAXONOMY.md. The canonical tests live under src/__Tests/** and each module’s __Tests/ directory.

Build parameters – values such as {{ dotnet }} (runtime) and {{ angular }} (UI framework) are injected at build time.

Test Philosophy

Core Principles

1. Determinism as Contract: Scan verdicts must be reproducible. Same inputs → byte-identical outputs.
2. Offline by Default: Every test (except explicitly tagged “online”) runs without network access.
3. Evidence-First Validation: Assertions verify the complete evidence chain, not just pass/fail.
4. Interop is Required: Compatibility with ecosystem tools (Syft, Grype, Trivy, cosign) blocks releases.
5. Coverage by Risk: Prioritize testing high-risk paths over line coverage metrics.

Test Boundaries

• Lattice/policy merge algorithms run in scanner.webservice
• Concelier/Excitors preserve prune source (no conflict resolution)
• Tests enforce these boundaries explicitly

Model taxonomy

See testing-strategy-models.mdfor the required test types per project model and the module-to-model mapping.

Layer Map

Test Categories (xUnit Traits)

[Trait("Category", "Unit")]           // Fast, isolated unit tests
[Trait("Category", "Property")]       // Property-based checks (sub-trait)
[Trait("Category", "Snapshot")]       // Golden/snapshot assertions (sub-trait)
[Trait("Category", "Integration")]    // Tests requiring infrastructure
[Trait("Category", "Contract")]       // Schema and API contract checks
[Trait("Category", "E2E")]            // Full end-to-end workflows
[Trait("Category", "AirGap")]         // Must work without network
[Trait("Category", "Interop")]        // Third-party tool compatibility
[Trait("Category", "Performance")]    // Performance benchmarks
[Trait("Category", "Chaos")]          // Failure injection tests
[Trait("Category", "Security")]       // Security-focused tests
[Trait("Category", "Live")]           // Opt-in upstream connector tests

Quality Gates

Local Runner

# minimal run: unit + property + frontend tests
./scripts/dev-test.sh

# full stack incl. Playwright and lighthouse
./scripts/dev-test.sh --full

# category-specific
dotnet test --filter "Category=Unit"
dotnet test --filter "Category=AirGap"
dotnet test --filter "Category=Interop"

The script spins up PostgreSQL/Valkey via Testcontainers and requires:

• Docker ≥ 25
• Node 20 (for the Angular/Vitest unit lane and Playwright)

PostgreSQL Testcontainers

Multiple suites (Concelier connectors, Excititor worker/WebService, Scheduler) use Testcontainers with PostgreSQL for integration tests. If you don’t have Docker available, tests can also run against a local PostgreSQL instance listening on 127.0.0.1:5432.

Local PostgreSQL Helper

Some suites (Concelier WebService/Core, Exporter JSON) need a full PostgreSQL instance when you want to debug or inspect data with psql. A helper script is available under tools/postgres/local-postgres.sh:

# start a local PostgreSQL instance
tools/postgres/local-postgres.sh start

# stop / clean
tools/postgres/local-postgres.sh stop
tools/postgres/local-postgres.sh clean

By default the script uses Docker to run PostgreSQL 16, binds to 127.0.0.1:5432, and creates a database called stellaops. The connection string is printed on start and you can export it before running dotnet test if a suite supports overriding its connection string.

New Test Infrastructure (Epic 5100)

Run Manifest & Replay

Every scan captures a Run Manifest containing all inputs (artifact digests, feed versions, policy versions, PRNG seed). This enables deterministic replay:

# Replay a scan from manifest
stella replay --manifest run-manifest.json --output verdict.json

# Verify determinism
stella replay verify --manifest run-manifest.json

Evidence Index

The Evidence Index links verdicts to their supporting evidence chain:

• Verdict → SBOM digests → Attestation IDs → Tool versions

Golden Corpus

Located at src/__Tests/__Datasets/golden-corpus/, contains 50+ test cases:

• Severity levels (Critical, High, Medium, Low)
• VEX scenarios (Not Affected, Affected, Conflicting)
• Reachability cases (Reachable, Not Reachable, Inconclusive)
• Unknowns scenarios
• Scale tests (200 to 50k+ packages)
• Multi-distro (Alpine, Debian, RHEL, SUSE, Ubuntu)
• Interop fixtures (Syft-generated, Trivy-generated)
• Negative cases (malformed inputs)

Offline Testing

Inherit from NetworkIsolatedTestBase for air-gap compliance:

[Trait("Category", "AirGap")]
public class OfflineTests : NetworkIsolatedTestBase
{
    [Fact]
    public async Task Test_WorksOffline()
    {
        // Test implementation
        AssertNoNetworkCalls();  // Fails if network accessed
    }
}

Concelier OSV↔GHSA Parity Fixtures

The Concelier connector suite includes a regression test (OsvGhsaParityRegressionTests) that checks a curated set of GHSA identifiers against OSV responses. The fixture snapshots live in src/Concelier/__Tests/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/ and are kept deterministic so the parity report remains reproducible.

To refresh the fixtures when GHSA/OSV payloads change:

1. Ensure outbound HTTPS access to https://api.osv.dev and https://api.github.com.
2. Run UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/__Tests/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj.
3. Commit the regenerated osv-ghsa.*.json files that the test emits (raw snapshots and canonical advisories).

The regen flow logs [Parity] messages and normalises recordedAt timestamps so the fixtures stay stable across machines.

CI Job Layout

flowchart LR
  subgraph fast-path
    U[xUnit] --> P[FsCheck] --> I1[Testcontainer API]
  end

  I1 --> FE[Jest]
  FE --> E2E[Playwright]
  E2E --> Lighthouse

  subgraph release-gates
    REPLAY[Replay Verify]
    INTEROP[Interop E2E]
    OFFLINE[Offline E2E]
    BUDGET[Unknowns Gate]
  end

  Lighthouse --> INTEG2[Concelier]
  INTEG2 --> LOAD[k6]
  LOAD --> CHAOS[Chaos Suite]
  CHAOS --> RELEASE[Attestation diff]

  RELEASE --> release-gates

Adding a New Test Layer

1. Extend scripts/dev-test.sh so local contributors get the layer by default.
2. Add a dedicated workflow in .gitea/workflows/.
3. Register the layer here (this document) and add its row to the test manifest (TEST_MANIFEST.yml); if it introduces a new quality gate, record the gate metric alongside the others in the Quality Gates table.
4. If the test requires network isolation, inherit from NetworkIsolatedTestBase.
5. If the test uses golden corpus, add cases to src/__Tests/__Datasets/golden-corpus/.

Related Documentation

• Testing README — complete testing documentation index
• Testing Strategy Models — required test types per project model
• Test Trait Taxonomy — canonical Category values and CI routing
• Test Suite Consolidation Deep Dive — source-derived suite inventory
• CI Lane Filters — how CI selects and gates by trait
• CI/CD Test Strategy — CI/CD integration details
• Offline Operation Guide
• Module Architecture Dossiers

Last updated 2026-05-31