Competitor Parity Testing Guide
Audience: engineers and product owners who need to confirm that the Stella Ops scanner keeps pace with industry-standard tools on coverage, accuracy, and performance — and who maintain the parity harness, fixtures, and drift alerts.
This document describes Stella Ops’ competitor parity testing methodology, which ensures the scanner maintains feature parity and performance competitiveness with industry-standard tools.
Overview
Parity testing compares Stella Ops against three primary competitors:
| Tool | Type | Primary Function |
|---|---|---|
| Syft (Anchore) | SBOM Generator | Package extraction and SBOM creation |
| Grype (Anchore) | Vulnerability Scanner | CVE detection using Syft SBOMs |
| Trivy (Aqua) | Multi-scanner | SBOM + vulnerability + secrets + misconfig |
Goals
- Prevent regression: Detect when StellaOps falls behind competitors on key metrics
- Track trends: Monitor parity metrics over time to identify drift patterns
- Guide development: Use competitor gaps to prioritize feature work
- Validate claims: Ensure marketing claims are backed by measurable data
Fixture Set
Tests run against a standardized set of container images that represent diverse workloads:
Quick Set (Nightly)
- Alpine 3.19
- Debian 12
- Ubuntu 24.04
- Python 3.12-slim
- Node.js 22-alpine
- nginx:1.25 (known vulnerabilities)
Full Set (Weekly)
All quick set images plus:
- RHEL 9 UBI
- Go 1.22-alpine
- Rust 1.79-slim
- OpenJDK 21-slim
- .NET 8.0-alpine
- postgres:16 (known vulnerabilities)
- wordpress:6.5 (complex application)
- valkey/valkey:7.2-alpine (Redis-compatible)
- Multi-layer image (test depth)
Metrics Collected
SBOM Metrics
| Metric | Description | Target |
|---|---|---|
| Package Count | Total packages detected | ≥95% of Syft |
| PURL Completeness | Packages with valid PURLs | ≥98% |
| License Detection | Packages with license info | ≥90% |
| CPE Mapping | Packages with CPE identifiers | ≥85% |
Vulnerability Metrics
| Metric | Description | Target |
|---|---|---|
| Recall | CVEs found vs. union of all scanners | ≥95% |
| Precision | True positives vs. total findings | ≥90% |
| F1 Score | Harmonic mean of precision/recall | ≥92% |
| Severity Distribution | Breakdown by Critical/High/Medium/Low | Match competitors ±10% |
Latency Metrics
| Metric | Description | Target |
|---|---|---|
| P50 | Median scan time | ≤1.5x Grype |
| P95 | 95th percentile scan time | ≤2x Grype |
| P99 | 99th percentile scan time | ≤3x Grype |
| Time-to-First-Signal | Time to first vulnerability found | ≤Grype |
Error Handling Metrics
| Scenario | Expected Behavior |
|---|---|
| Malformed manifest | Graceful degradation with partial results |
| Network timeout | Clear error message; cached feed fallback |
| Large image (>5GB) | Streaming extraction; no OOM |
| Corrupt layer | Skip layer; report warning |
| Missing base layer | Report incomplete scan |
| Registry auth failure | Clear auth error; suggest remediation |
| Rate limiting | Backoff + retry; clear message |
Running Parity Tests
Locally
# Build the parity test project
dotnet build src/__Tests/parity/StellaOps.Parity.Tests
# Run quick fixture set
dotnet test src/__Tests/parity/StellaOps.Parity.Tests \
-e PARITY_FIXTURE_SET=quick \
-e PARITY_OUTPUT_PATH=./parity-results
# Run full fixture set
dotnet test src/__Tests/parity/StellaOps.Parity.Tests \
-e PARITY_FIXTURE_SET=full \
-e PARITY_OUTPUT_PATH=./parity-results
Prerequisites
Ensure competitor tools are installed and in PATH:
# Install Syft
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin v1.9.0
# Install Grype
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin v0.79.3
# Install Trivy
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin v0.54.1
CI/CD
The dedicated parity workflows (parity-tests.yml, competitor-parity.yml, benchmark-vs-competitors.yml) have been archived to .gitea/workflows-archived/; competitor benchmarking currently runs through .gitea/workflows/dotnet-benchmarks.yml. The intended cadence is:
- Nightly (02:00 UTC): quick fixture set
- Weekly (Sunday 00:00 UTC): full fixture set
Results are stored as workflow artifacts and optionally pushed to Prometheus.
Drift Detection
The parity system includes automated drift detection that alerts when StellaOps falls behind competitors:
Thresholds
| Metric | Threshold | Trend Period |
|---|---|---|
| SBOM Completeness | >5% decline | 3 days |
| Vulnerability Recall | >5% decline | 3 days |
| Latency vs Grype | >10% increase | 3 days |
| PURL Completeness | >5% decline | 3 days |
| F1 Score | >5% decline | 3 days |
Alert Severity
| Severity | Condition | Action |
|---|---|---|
| Low | 1-1.5x threshold | Monitor |
| Medium | 1.5-2x threshold | Investigate within sprint |
| High | 2-3x threshold | Prioritize fix |
| Critical | >3x threshold | Immediate action required |
Analyzing Drift
# Run drift analysis on stored results
dotnet run --project src/__Tests/parity/StellaOps.Parity.Tests \
-- analyze-drift \
--results-path ./parity-results \
--threshold 0.05 \
--trend-days 3
Result Storage
JSON Format
Results are stored as timestamped JSON files:
parity-results/
├── parity-20250115T020000Z-abc123.json
├── parity-20250116T020000Z-def456.json
└── parity-20250122T000000Z-ghi789.json # Weekly
Retention Policy
- Last 90 days: Full detail retained
- Older than 90 days: Aggregated to weekly summaries
- Artifacts: Workflow artifacts retained for 90 days
Prometheus Export
Results can be exported to Prometheus for dashboarding:
stellaops_parity_sbom_completeness_ratio{run_id="..."} 0.97
stellaops_parity_vuln_recall{run_id="..."} 0.95
stellaops_parity_latency_p95_ms{scanner="stellaops",run_id="..."} 1250
InfluxDB Export
For InfluxDB time-series storage:
parity_sbom,run_id=abc123 completeness_ratio=0.97,matched_count=142i 1705280400000000000
parity_vuln,run_id=abc123 recall=0.95,precision=0.92,f1=0.935 1705280400000000000
Competitor Version Tracking
Competitor tools are pinned to specific versions to ensure reproducibility:
| Tool | Current Version | Last Updated |
|---|---|---|
| Syft | 1.9.0 | 2025-01-15 |
| Grype | 0.79.3 | 2025-01-15 |
| Trivy | 0.54.1 | 2025-01-15 |
Updating Versions
- Update the pinned version in the parity workflow (currently
.gitea/workflows-archived/parity-tests.yml) - Update the version constants in
src/__Tests/parity/StellaOps.Parity.Tests/ParityHarness.cs - Run the full parity test to establish a new baseline
- Document the version change in the sprint execution log
Troubleshooting
Common Issues
Syft not found
Error: Syft executable not found in PATH
Solution: Install Syft or set SYFT_PATH environment variable.
Grype DB outdated
Warning: Grype vulnerability database is 7+ days old
Solution: Run grype db update to refresh the database.
Image pull rate limit
Error: docker pull rate limit exceeded
Solution: Use authenticated Docker Hub credentials or local registry mirror.
Test timeout
Error: Test exceeded 30 minute timeout
Solution: Increase timeout or use quick fixture set.
Debug Mode
Enable verbose logging:
dotnet test src/__Tests/parity/StellaOps.Parity.Tests \
-e PARITY_DEBUG=true \
-e PARITY_KEEP_OUTPUTS=true
Architecture
src/__Tests/parity/StellaOps.Parity.Tests/
├── StellaOps.Parity.Tests.csproj # Project file
├── ParityTestFixtureSet.cs # Container image fixtures
├── ParityHarness.cs # Scanner execution harness
├── SbomComparisonLogic.cs # SBOM comparison
├── VulnerabilityComparisonLogic.cs # Vulnerability comparison
├── LatencyComparisonLogic.cs # Latency comparison
├── ErrorModeComparisonLogic.cs # Error handling comparison
└── Storage/
├── ParityResultStore.cs # Time-series storage
└── ParityDriftDetector.cs # Drift detection
See Also
- Scanner Architecture
- Test Suite Overview
- CI Quality Gates
- Live benchmark workflow:
.gitea/workflows/dotnet-benchmarks.yml - Archived parity workflows:
.gitea/workflows-archived/parity-tests.yml,.gitea/workflows-archived/benchmark-vs-competitors.yml