SBOM Interoperability Testing

This guide documents the interoperability test suite that proves Stella Ops SBOMs work with the wider supply-chain security ecosystem. It is written for QA engineers and contributors who run, extend, or debug those tests.

Overview

Stella Ops SBOM interoperability tests ensure compatibility with third-party security tools across the ecosystem. They validate that Stella Ops-generated SBOMs can be consumed by popular tools such as Grype, and that vulnerability findings parity stays at or above 95%.

Test Coverage

SBOM Formats

Notes:

• SPDX 3.0.1 generation currently emits JSON-LD @context, spdxVersion, core document/package/relationship elements (including agent/tool elements for creationInfo), software package/file/snippet metadata, build profile elements with output relationships, security vulnerabilities with assessment relationships, licensing license elements with declared/concluded relationships, AI AIPackage metadata (autonomy, domain, metrics, safety risk assessment), Dataset package metadata (type, collection, preprocessing, availability), verifiedUsing hashes/signatures, external references/identifiers (including externalRef contentType when available), namespaceMap/imports for cross-document references, extension metadata via SbomExtension namespace/properties on document/component/vulnerability elements, and Lite profile output (opt-in via SpdxWriterOptions.UseLiteProfile). Full profile coverage is tracked in SPRINT_20260119_014.

Third-Party Tools

Parity Expectations

What is Parity?

Parity measures how closely Stella Ops vulnerability findings match those from third-party tools like Grype when scanning the same SBOM.

Formula:

Parity % = (Matching Findings / Total Unique Findings) × 100

Target: ≥95% parity for both CycloneDX and SPDX formats

Known Differences

The following differences are acceptable and expected:

1. VEX Application

• Difference: Stella Ops applies VEX documents; Grype may not
• Impact: Stella Ops may show fewer vulnerabilities
• Acceptable: Yes - this is a feature, not a bug

2. Feed Coverage

• Difference: Tool-specific vulnerability databases
• Examples:
  • Stella Ops may have distro-specific feeds Grype lacks
  • Grype may have GitHub Advisory feeds Stella Ops does not prioritize
• Acceptable: Within 5% tolerance

3. Version Matching Semantics

• Difference: Interpretation of version ranges
• Examples:
  • SemVer vs non-SemVer handling
  • Epoch handling in RPM/Debian packages
• Acceptable: When using distro-native comparators

4. Package Identification (PURL)

• Difference: PURL generation strategies
• Examples:
  • pkg:npm/package vs pkg:npm/package@version
  • Namespace handling
• Acceptable: When functionally equivalent

Running Interop Tests

Prerequisites

Install required tools:

# Install Syft
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin

# Install Grype
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin

# Install cosign
curl -sSfL https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 -o /usr/local/bin/cosign
chmod +x /usr/local/bin/cosign

Local Execution

# Run all interop tests
dotnet test tests/interop/StellaOps.Interop.Tests

# Run CycloneDX tests only
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Format=CycloneDX"

# Run SPDX tests only
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Format=SPDX"

# Run parity tests
dotnet test tests/interop/StellaOps.Interop.Tests --filter "Category=Parity"

CI Execution

Interop tests run automatically on:

• Pull requests affecting scanner or SBOM code
• Nightly schedule (6 AM UTC)
• Manual workflow dispatch

See .gitea/workflows/interop-e2e.yml for CI configuration.

Test Images

The following container images are used for interop testing:

Troubleshooting

Parity Below Threshold

If parity drops below 95%:

1. Check for feed updates

   • Grype may have newer vulnerability data
   • Update Stella Ops feeds

2. Review differences

   • Run parity analysis: dotnet test --filter "Category=Parity" --logger "console;verbosity=detailed"
   • Categorize differences using FindingsParityAnalyzer

3. Validate with golden corpus

   • Compare against known-good results in src/__Tests/__Datasets/golden-corpus/categories/interop/

4. Update acceptable differences

   • Document new acceptable differences in this README
   • Adjust tolerance if justified

Tool Installation Failures

If Syft/Grype/cosign fail to install:

# Check versions
syft --version
grype --version
cosign version

# Reinstall if needed
rm /usr/local/bin/{syft,grype,cosign}
# Re-run installation commands

SBOM Validation Failures

If SBOMs fail schema validation:

1. Verify format version:

   jq '.specVersion' sbom-cyclonedx.json  # Should be "1.7"
   jq '.spdxVersion' sbom-spdx.json       # Should be "SPDX-3.0"
   

2. Validate against official schemas:

   # CycloneDX
   npm install -g @cyclonedx/cdx-cli
   cdx-cli validate --input-file sbom-cyclonedx.json
   
   # SPDX: validate with the SPDX online validation tools
   # (see https://tools.spdx.org/).
   

Continuous Improvement

Adding New Test Cases

1. Add new image to test matrix in *RoundTripTests.cs
2. Update TestImages member data
3. Run locally to verify
4. Submit PR with updated tests

Updating Parity Thresholds

Current threshold: 95%

To adjust:

1. Document justification in sprint file
2. Update tolerancePercent parameter in test calls
3. Update this README

Tool Version Pinning

Tools are currently installed from latest. To pin versions:

1. Update .gitea/workflows/interop-e2e.yml
2. Specify version in install commands
3. Document version compatibility in this README

References

• CycloneDX 1.7 Specification
• SPDX 3.0.1 Specification
• Syft Documentation
• Grype Documentation
• cosign Documentation

Contacts

For questions about interop testing:

• Sprint: SPRINT_5100_0003_0001
• Owner: QA Team
• Dependencies: Sprint 5100.0001.0002 (Evidence Index)