VexLens Testing Strategy

Overview

This document describes the testing strategy for VexLens module, covering determinism verification, regression testing, and golden corpus validation.

Test Categories

1. Unit Tests (Category=Unit)

Standard unit tests for individual components:

2. Determinism Tests (Category=Determinism)

Tests that verify VexLens produces identical outputs for identical inputs:

Known Limitations

The VexProofBuilder.GenerateProofId() method currently uses Guid.NewGuid() which introduces non-determinism in the ProofId (and consequently the Digest). This is tracked as risk R-003 and violates AGENTS.md Rule 8.2 (Inject TimeProvider / ID generators). Until IGuidGenerator injection is added, determinism tests validate structural determinism rather than byte-identical outputs.

3. Regression Tests (Category=Regression)

Tests that validate known scenarios produce expected verdicts:

4. Golden Corpus Tests (Category=Golden)

Tests using curated real-world backport scenarios:

Test Infrastructure

Golden Corpus Structure

src/__Tests/__Datasets/GoldenBackports/
├── index.json              # Corpus index with case metadata
├── CVE-YYYY-XXXXX-distro-package/
│   └── case.json           # Individual test case
└── ...

Case Schema

{
  "id": "case identifier",
  "cve": "CVE-YYYY-XXXXX",
  "distro": {
    "name": "debian|rhel|...",
    "version": "7|8|...",
    "family": "debian|rpm|..."
  },
  "package": {
    "name": "package-name",
    "binary": "binary-name",
    "vulnerableEvr": "vulnerable EVR",
    "patchedEvr": "patched EVR"
  },
  "upstream": {
    "vulnerableRange": "SemVer constraint",
    "fixedVersion": "upstream fix version"
  },
  "expectedVerdict": {
    "status": "fixed|not_affected|affected",
    "reason": "backport_detected|...",
    "upstreamWouldSay": "affected|..."
  },
  "evidence": {
    "advisoryUrl": "URL to advisory",
    "patchCommit": "optional commit hash",
    "notes": "explanation"
  }
}

Test Runner Components

  1. GoldenCorpusLoader - Loads and filters corpus cases
  2. GoldenCorpusTestRunner - Executes evaluator against corpus
  3. GoldenCorpusTests - xUnit test class for corpus validation

CI Integration

Nightly Regression (.gitea/workflows/nightly-regression.yml)

Test categories run nightly:

PR Gating

Quick tests run on every PR:

Test Best Practices

1. Use FakeTimeProvider

Always inject Microsoft.Extensions.Time.Testing.FakeTimeProvider to ensure deterministic timestamps:

private readonly FakeTimeProvider _timeProvider;
private readonly DateTimeOffset _fixedTime = new(2026, 1, 3, 12, 0, 0, TimeSpan.Zero);

public MyTests()
{
    _timeProvider = new FakeTimeProvider(_fixedTime);
}

2. Set Confidence Explicitly

The VexProofBuilder calculates confidence from explicit weight values. For predictable test assertions, set all confidence-affecting properties:

builder
    .WithWeightSpread(0.95m)
    .WithConditionCoverage(1.0m)
    .WithSignatureBonus(0.10m)
    .WithFreshnessBonus(0.05m);

3. Set Justification with Status

When testing not_affected status, pass justification to WithFinalStatus:

.WithFinalStatus(VexStatus.NotAffected, VexJustification.VulnerableCodeNotPresent)

4. Use Correct Types

The AddStatement method requires proper types:

builder.AddStatement(
    id: "stmt-001",
    source: "vendor-csaf",
    issuer: new VexProofIssuer("Vendor", IssuerCategory.Vendor, TrustTier.Authoritative),
    status: VexStatus.Fixed,
    justification: null,
    weight: new VexProofWeight(0.95m, new VexProofWeightFactors(0.95m, 1.0m, 0.9m, 1.0m, 0.8m)),
    timestamp: _fixedTime,
    signatureVerified: true);

5. IssuerCategory and TrustTier Values

Valid IssuerCategory values:

Valid TrustTier values:

Future Improvements

  1. Inject IGuidGenerator - Enable full digest determinism (R-003)
  2. Expand golden corpus - Add more edge cases and distributions
  3. Property-based testing - Use FsCheck for fuzzing VEX inputs
  4. Mutation testing - Use Stryker.NET to validate test coverage quality