Automated Test‑Suite Overview

This document enumerates every automated check executed by the Stella Ops CI pipeline, from unit level to chaos experiments. It is intended for contributors who need to extend coverage or diagnose failures.

Build parameters – values such as {{ dotnet }} (runtime) and {{ angular }} (UI framework) are injected at build time.

Layer map

Layer	Tooling	Entry‑point	Frequency
1. Unit	`xUnit` (`dotnet test`)	`*.Tests.csproj`	per PR / push
2. Property‑based	`FsCheck`	`SbomPropertyTests`	per PR
3. Integration (API)	`Testcontainers` suite	`test/Api.Integration`	per PR + nightly
4. Integration (DB-merge)	in-memory Mongo + Redis	`Concelier.Integration` (vulnerability ingest/merge/export service)	per PR
5. Contract (gRPC)	`Buf breaking`	`buf.yaml` files	per PR
6. Front‑end unit	`Jest`	`ui/src/*/.spec.ts`	per PR
7. Front‑end E2E	`Playwright`	`ui/e2e/**`	nightly
8. Lighthouse perf / a11y	`lighthouse-ci` (Chrome headless)	`ui/dist/index.html`	nightly
9. Load	`k6` scripted scenarios	`k6/*.js`	nightly
10. Chaos CPU / OOM	`pumba`	Docker Compose overlay	weekly
11. Dependency scanning	`Trivy fs` + `dotnet list package --vuln`	root	per PR
12. License compliance	`LicenceFinder`	root	per PR
13. SBOM reproducibility	`in‑toto attestation` diff	GitLab job	release tags

Quality gates

Metric	Budget	Gate
API unit coverage	≥ 85 % lines	PR merge
API response P95	≤ 120 ms	nightly alert
Δ‑SBOM warm scan P95 (4 vCPU)	≤ 5 s	nightly alert
Lighthouse performance score	≥ 90	nightly alert
Lighthouse accessibility score	≥ 95	nightly alert
k6 sustained RPS drop	< 5 % vs baseline	nightly alert

Local runner

# minimal run: unit + property + frontend tests
./scripts/dev-test.sh

# full stack incl. Playwright and lighthouse
./scripts/dev-test.sh --full

The script spins up MongoDB/Redis via Testcontainers and requires:

Docker ≥ 25
Node 20 (for Jest/Playwright)

Mongo2Go / OpenSSL shim

Multiple suites (Concelier connectors, Excititor worker/WebService, Scheduler) fall back to Mongo2Go when a developer does not have a local mongod listening on 127.0.0.1:27017. This is a test-only dependency: production/dev runtime MongoDB always runs inside the compose/k8s network using the standard StellaOps cryptography stack. Modern distros ship OpenSSL 3 by default, so when Mongo2Go starts its embedded mongod you must expose the legacy OpenSSL 1.1 libraries that binary expects:

From the repo root, export the provided binaries before running any tests:

export LD_LIBRARY_PATH="$(pwd)/tests/native/openssl-1.1/linux-x64:${LD_LIBRARY_PATH:-}"

(Optional) If you only need the shim for a single command, prefix it:

LD_LIBRARY_PATH="$(pwd)/tests/native/openssl-1.1/linux-x64" \
  dotnet test src/Concelier/StellaOps.Concelier.sln --nologo

CI runners or dev containers should either copy tests/native/openssl-1.1/linux-x64/libcrypto.so.1.1 and libssl.so.1.1 into a directory that is already on the default library path, or export the LD_LIBRARY_PATH value shown above before invoking dotnet test.

The shim lives under tests/native/openssl-1.1/README.md with upstream source and licensing details. When the system already has OpenSSL 1.1 installed you can skip this step.

Local Mongo helper

Some suites (Concelier WebService/Core, Exporter JSON) need a full mongod instance when you want to debug outside of Mongo2Go (for example to inspect data with mongosh or pin a specific server version). A thin wrapper is available under tools/mongodb/local-mongo.sh:

# download (cached under .cache/mongodb-local) and start a local replica set
tools/mongodb/local-mongo.sh start

# reuse an existing data set
tools/mongodb/local-mongo.sh restart

# stop / clean
tools/mongodb/local-mongo.sh stop
tools/mongodb/local-mongo.sh clean

By default the script downloads MongoDB 6.0.16 for Ubuntu 22.04, binds to 127.0.0.1:27017, and initialises a single-node replica set called rs0. The current URI is printed on start, e.g. mongodb://127.0.0.1:27017/?replicaSet=rs0, and you can export it before running dotnet test if a suite supports overriding its connection string.

Concelier OSV↔GHSA parity fixtures

The Concelier connector suite includes a regression test (OsvGhsaParityRegressionTests) that checks a curated set of GHSA identifiers against OSV responses. The fixture snapshots live in src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/Fixtures/ and are kept deterministic so the parity report remains reproducible.

To refresh the fixtures when GHSA/OSV payloads change:

Ensure outbound HTTPS access to https://api.osv.dev and https://api.github.com.
Run UPDATE_PARITY_FIXTURES=1 dotnet test src/Concelier/StellaOps.Concelier.PluginBinaries/StellaOps.Concelier.Connector.Osv.Tests/StellaOps.Concelier.Connector.Osv.Tests.csproj.
Commit the regenerated osv-ghsa.*.json files that the test emits (raw snapshots and canonical advisories).

The regen flow logs [Parity] messages and normalises recordedAt timestamps so the fixtures stay stable across machines.

CI job layout

flowchart LR
  subgraph fast-path
    U[xUnit] --> P[FsCheck] --> I1[Testcontainer API]
  end

  I1 --> FE[Jest]
  FE --> E2E[Playwright]
  E2E --> Lighthouse
  Lighthouse --> INTEG2[Concelier]
  INTEG2 --> LOAD[k6]
  LOAD --> CHAOS[pumba]
  CHAOS --> RELEASE[Attestation diff]

Adding a new test layer

Extend scripts/dev-test.sh so local contributors get the layer by default.
Add a dedicated GitLab job in .gitlab-ci.yml (stage test or nightly).
Register the job in docs/19_TEST_SUITE_OVERVIEW.md and list its metric in docs/metrics/README.md.

Last updated {{ “now” | date: “%Y‑%m‑%d” }}