19 · Test‑Suite Overview — Stella Ops

(v2.0 — 12 Jul 2025)

Purpose — Describe the multi‑layer automated‑test strategy that guards Stella Ops’ five‑second performance promise, security posture and API stability, and show how each layer maps to CI gates and release criteria.

0 Table of Contents

1. Test‑pyramid at a glance
2. Layer definitions & tooling
3. Directory & naming conventions
4. CI workflows & failure policy
5. Quality gates & coverage budgets
6. Evidence retention & auditability
7. Local developer quick‑start
8. Flaky‑test triage & escalation
9. Change log

1 Test‑pyramid at a glance

2 Layer definitions & tooling

2.1 Unit

• Target ≥ 80 % line and ≥ 60 % branch coverage (coverlet + ReportGenerator).
• Naming: Method_ShouldExpected_WhenCondition.

2.2 Mutation

• Stryker.NET runs only on projects tagged critical‑logic=true in Directory.Build.props.
• Threshold: ≥ 60 % mutation score; red build < 55 %.

2.3 Integration

• RedisTestcontainer, TrivyServerTestcontainer, TestcontainersNetwork for realistic wiring.

• Each test cleans keys and volumes; parallelisable.

• Quota & throttle tests (new) — spin up Redis container, fix system clock to just before UTC midnight, hammer /scan with a stub token to validate:

  1. Counter hits 200 → header X‑Stella‑Quota‑Remaining: 133; banner socket event emitted. Delay of 5 secs is added.
  2. Counter hits 333 → Delay of 60 secs is added.
  3. At UTC midnight rollover key expires → counter resets to 0.

### 2.4 Quota / throttle layer (explicit)

• Uses the same fixture but runs in isolation to keep CI time predictable.
• Fails the pipeline if any of the four behaviours above mis‑fires.

2.4 End‑to‑End

• API suite asserts presence of X‑Stella‑Quota‑Remaining on every successful /scan.
• API suite uses async httpx for accurate latency numbers.
• UI suite uses Playwright headless Chromium; Lighthouse a11y snapshot recorded.

2.5 Performance

• Hyperfine measures CLI workflows (SBOM_LOCAL, SBOM_REMOTE, IMAGE_WARM).
• K6 hits /scan at 40 rps for 3 min; checks P95 ≤ 5 s and error‑rate = 0.

2.6 Security (DAST + SAST)

• PHASE QUOTA_WAIT benchmark:
  • ≤ 5 s median for first 30 blocked requests (soft back‑off).
  • Exactly 60 s wall for hard wait‑wall.
• SAST: CodeQL (GitHub native) + Semgrep OSS ruleset.
• DAST: ZAP baseline spider + passive rules; fails on High risk alerts.

2.7 Chaos / Resilience

• Pumba randomly kills Trivy side‑car; test asserts queue retry.
• Toxiproxy injects 150 ms latency on Redis; perf budget still ≤ 6 s.

3 Repository layout

tests/
├─ unit/                 # *.Unit.csproj
├─ mutation/stryker.conf.json
├─ integration/          # *.Integration.csproj
│   └─ fixtures/
├─ e2e/
│   ├─ api/pytest/       # test_*.py
│   └─ ui/playwright/    # *.spec.ts
├─ perf/
│   ├─ compose-perf.yml
│   ├─ hyperfine/
│   └─ k6/
├─ security/
│   ├─ zap-baseline.conf
│   └─ semgrep/
└─ chaos/
    ├─ toxiproxy/
    └─ pumba/

Tests mirror the module namespaces; each src project owns a matching test project.

4 CI workflows

5 Quality gates & budgets

Coverage & perf budgets live in tests/budgets/*.json; CI actions fail on regression.

6 Evidence retention

Test logs (JUnit/Allure) 12 months S3, lifecycle policy

7 Developer quick‑start

Bring up full stack for e2e on a laptop

docker compose -f tests/e2e/compose-core.yml up -d

Run unit + integration

dotnet test --collect:"XPlat Code Coverage"

# API e2e
cd tests/e2e/api
pytest -q

# UI e2e
cd tests/e2e/ui
npx playwright install
npm test

8 Flaky‑test triage & escalation

Label failing test with flaky and open GitHub Discussion. After 3 consecutive nightly failures, auto‑page ops@stella-ops.org. Root‑cause within next sprint or quarantine behind feature flag (max 2 weeks). Token‑expiry tests cannot be quarantined — they guard offline operability.

9 Change log

(End of Test‑Suite Overview v2.0)