Testing Quality Guardrails Implementation

Audience: test-automation leads and the platform team planning CI quality-gate work across Stella Ops.

Overview

This document is the master implementation plan for the Testing Quality Guardrails system, derived from the Testing and Quality Guardrails Technical Reference product advisory.

Executive Summary

This implementation closes the gaps between the product advisory and the current Stella Ops codebase. After analysis, we identified six high-value items to implement across four focused sprints.

What We’re Implementing

What We’re NOT Implementing (and Why)

Sprint Roadmap

┌─────────────────────────────────────────────────────────────────┐
│                    TESTING QUALITY GUARDRAILS                    │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Sprint 0350                Sprint 0351                          │
│  CI Quality Gates           SCA Failure Catalogue                │
│  ┌─────────────┐            ┌─────────────┐                     │
│  │ Reachability│            │ FC6: Java   │                     │
│  │ TTFS        │            │ FC7: .NET   │                     │
│  │ Performance │            │ FC8: Docker │                     │
│  └─────────────┘            │ FC9: PURL   │                     │
│       │                     │ FC10: CVE   │                     │
│       │                     └─────────────┘                     │
│       │                          │                              │
│       └──────────┬───────────────┘                              │
│                  │                                               │
│                  ▼                                               │
│  Sprint 0352                Sprint 0353                          │
│  Security Testing           Mutation Testing                     │
│  ┌─────────────┐            ┌─────────────┐                     │
│  │ OWASP Top 10│            │ Stryker.NET │                     │
│  │ A01-A10     │            │ Scanner     │                     │
│  │ 50+ tests   │            │ Policy      │                     │
│  └─────────────┘            │ Authority   │                     │
│                             └─────────────┘                     │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Execution Order

1. Sprint 0350 and Sprint 0351 can run in parallel (no dependencies)
2. Sprint 0352 can run in parallel with 0350/0351
3. Sprint 0353 should start after 0352 (security tests should be stable first)

Estimated Duration

Sprint Details

Sprint 0350: CI Quality Gates Foundation

File: docs/implplan/SPRINT_0350_0001_0001_ci_quality_gates_foundation.md

Objective: Connect existing test infrastructure to CI enforcement

Key Deliverables:

• scripts/ci/compute-reachability-metrics.sh - Compute recall/precision from corpus
• scripts/ci/reachability-thresholds.yaml - Enforcement thresholds
• scripts/ci/compute-ttfs-metrics.sh - TTFS extraction from test runs
• src/__Tests/__Benchmarks/baselines/ttfs-baseline.json - TTFS targets
• scripts/ci/enforce-performance-slos.sh - Performance SLO checks
• CI workflow modifications for quality gates

Quality Thresholds:

thresholds:
  runtime_dependency_recall: >= 0.95
  unreachable_false_positives: <= 0.05
  reachability_underreport: <= 0.10
  ttfs_regression: <= +10% vs main

Sprint 0351: SCA Failure Catalogue Completion

File: docs/implplan/SPRINT_0351_0001_0001_sca_failure_catalogue_completion.md

Objective: Complete FC6-FC10 test cases for scanner regression testing

New Failure Cases:

Key Deliverables:

• 5 new fixture directories under src/__Tests/fixtures/sca/catalogue/
• DSSE manifests for integrity verification
• xUnit test project for failure catalogue
• Updated documentation

Sprint 0352: Security Testing Framework

File: docs/implplan/SPRINT_0352_0001_0001_security_testing_framework.md

Objective: Systematic OWASP Top 10 coverage for Stella Ops

Coverage Matrix:

Key Deliverables:

• tests/security/StellaOps.Security.Tests/ - Security test project
• MaliciousPayloads.cs - Common attack patterns
• SecurityTestBase.cs - Test infrastructure
• .gitea/workflows/security-tests.yml - Dedicated CI workflow
• docs/technical/testing/security-testing-guide.md - Documentation

Sprint 0353: Mutation Testing Integration

File: docs/implplan/SPRINT_0353_0001_0001_mutation_testing_integration.md

Objective: Measure test suite effectiveness with Stryker.NET

Target Modules:

Key Deliverables:

• Stryker.NET configuration for each target module
• scripts/ci/mutation-thresholds.yaml - Threshold configuration
• .gitea/workflows/mutation-testing.yml - Weekly mutation runs
• src/__Tests/__Benchmarks/baselines/mutation-baselines.json - Baseline scores
• docs/technical/testing/mutation-testing-guide.md - Developer guide

Existing Infrastructure Mapping

The advisory describes structures that already exist under different names:

Quality Gate Summary

After implementation, CI will enforce:

Reachability Gates

• Runtime dependency recall ≥ 95%
• Unreachable false positives ≤ 5%
• Reachability underreport ≤ 10%

Performance Gates

• Medium service scan < 2 minutes
• Reachability compute < 30 seconds
• SBOM ingestion < 5 seconds

TTFS Gates

• p50 < 2 seconds
• p95 < 5 seconds
• Regression ≤ +10% vs main

Coverage Gates

• Line coverage ≥ 70% (existing)
• Branch coverage ≥ 60% (existing)
• Mutation score ≥ 60-65% (break threshold)

Security Gates

• All security tests pass
• No OWASP Top 10 violations

File Changes Summary

New Files

scripts/ci/
├── compute-reachability-metrics.sh
├── compute-ttfs-metrics.sh
├── enforce-performance-slos.sh
├── enforce-thresholds.sh
├── enforce-mutation-thresholds.sh
├── extract-mutation-score.sh
├── reachability-thresholds.yaml
└── mutation-thresholds.yaml

src/__Tests/__Benchmarks/baselines/
├── ttfs-baseline.json
└── mutation-baselines.json

tests/
├── fixtures/sca/catalogue/
│   ├── fc6-java-shadow-jar/
│   ├── fc7-dotnet-transitive-pinning/
│   ├── fc8-docker-multistage-leakage/
│   ├── fc9-purl-namespace-collision/
│   └── fc10-cve-split-merge/
└── security/
    └── StellaOps.Security.Tests/

.gitea/workflows/
├── security-tests.yml
└── mutation-testing.yml

.config/
└── dotnet-tools.json (stryker)

stryker-config.json (root)

src/Scanner/__Libraries/StellaOps.Scanner.Core/stryker-config.json
src/Policy/StellaOps.Policy.Engine/stryker-config.json
src/Authority/StellaOps.Authority.Core/stryker-config.json

docs/technical/testing/
├── ci-quality-gates.md
├── security-testing-guide.md
└── mutation-testing-guide.md

Modified Files

.gitea/workflows/build-test-deploy.yml
src/__Tests/fixtures/sca/catalogue/inputs.lock
src/__Tests/fixtures/sca/catalogue/README.md
README.md (badges)

Rollback Strategy

If quality gates cause CI instability:

1. Immediate: Set failure_mode: warn in threshold configs
2. Short-term: Remove needs: dependencies to unblock other jobs
3. Investigation: Create issue with specific threshold that failed
4. Resolution: Either fix underlying issue or adjust threshold
5. Re-enable: Set failure_mode: block after verification

Success Metrics

References

Sprint Files

• docs/implplan/SPRINT_0350_0001_0001_ci_quality_gates_foundation.md
• docs/implplan/SPRINT_0351_0001_0001_sca_failure_catalogue_completion.md
• docs/implplan/SPRINT_0352_0001_0001_security_testing_framework.md
• docs/implplan/SPRINT_0353_0001_0001_mutation_testing_integration.md

Source Advisory

• docs/product/advisories/14-Dec-2025 - Testing and Quality Guardrails Technical Reference.md

Existing Documentation

• Test suite overview
• Reach-graph ground-truth schema
• Reach-graph corpus plan
• src/__Tests/reachability/README.md

Document Version