End-to-End Reproducibility Testing Guide

Audience: engineers maintaining the full-pipeline reproducibility suite, and reviewers diagnosing why two pipeline runs produced different hashes.

This guide covers whole-pipeline reproducibility — proving that the same inputs yield the same verdict, bundle, and envelope across runs, threads, and platforms. For unit-level determinism assertions see Determinism Gates; for the artifact manifest/baseline CI gate see Determinism Verification.

Sprint: SPRINT_8200_0001_0004_e2e_reproducibility_test Tasks: E2E-8200-025, E2E-8200-026

Overview

Stella Ops implements comprehensive end-to-end (E2E) reproducibility testing to ensure that identical inputs always produce identical outputs across:

• Sequential pipeline runs
• Parallel pipeline runs
• Different execution environments (Ubuntu, Windows, macOS)
• Different points in time (using frozen timestamps)

This document describes the E2E test structure, how to run tests, and how to troubleshoot reproducibility failures.

Test Architecture

Pipeline Stages

The E2E reproducibility tests cover the full security scanning pipeline:

┌─────────────────────────────────────────────────────────────────────────────┐
│                        Full E2E Pipeline                                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌──────────┐    ┌───────────┐    ┌──────┐    ┌────────┐    ┌──────────┐   │
│  │  Ingest  │───▶│ Normalize │───▶│ Diff │───▶│ Decide │───▶│  Attest  │   │
│  │ Advisory │    │  Merge &  │    │ SBOM │    │ Policy │    │   DSSE   │   │
│  │  Feeds   │    │  Dedup    │    │  vs  │    │ Verdict│    │ Envelope │   │
│  └──────────┘    └───────────┘    │Adviso│    └────────┘    └──────────┘   │
│                                   │ries  │                        │         │
│                                   └──────┘                        ▼         │
│                                                             ┌──────────┐    │
│                                                             │  Bundle  │    │
│                                                             │ Package  │    │
│                                                             └──────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘

Key Components

Running E2E Tests

Prerequisites

• .NET 10.0 SDK
• Docker (for PostgreSQL container)
• At least 4GB RAM available

Local Execution

# Run all E2E reproducibility tests
dotnet test src/__Tests/Integration/StellaOps.Integration.E2E/ \
  --logger "console;verbosity=detailed"

# Run specific test category
dotnet test src/__Tests/Integration/StellaOps.Integration.E2E/ \
  --filter "Category=Integration" \
  --logger "console;verbosity=detailed"

# Run with code coverage
dotnet test src/__Tests/Integration/StellaOps.Integration.E2E/ \
  --collect:"XPlat Code Coverage" \
  --results-directory ./TestResults

CI Execution

The dedicated E2E reproducibility workflow has been archived to .gitea/workflows-archived/e2e-reproducibility.yml. As designed, it runs on:

• Pull requests affecting src/**
• Pushes to main and develop branches
• Nightly at 2:00 AM UTC (full cross-platform suite)
• Manual trigger with optional cross-platform flag

Test Categories

1. Sequential Reproducibility (Tasks 11-14)

Tests that the pipeline produces identical results when run multiple times:

[Fact]
public async Task FullPipeline_ProducesIdenticalVerdictHash_AcrossRuns()
{
    // Arrange
    var inputs = await _fixture.SnapshotInputsAsync();

    // Act - Run twice
    var result1 = await _fixture.RunFullPipelineAsync(inputs);
    var result2 = await _fixture.RunFullPipelineAsync(inputs);

    // Assert
    result1.VerdictId.Should().Be(result2.VerdictId);
    result1.BundleManifestHash.Should().Be(result2.BundleManifestHash);
}

2. Parallel Reproducibility (Task 14)

Tests that concurrent execution produces identical results:

[Fact]
public async Task FullPipeline_ParallelExecution_10Concurrent_AllIdentical()
{
    var inputs = await _fixture.SnapshotInputsAsync();
    const int concurrentRuns = 10;

    var tasks = Enumerable.Range(0, concurrentRuns)
        .Select(_ => _fixture.RunFullPipelineAsync(inputs));

    var results = await Task.WhenAll(tasks);
    var comparison = ManifestComparer.CompareMultiple(results.ToList());
    
    comparison.AllMatch.Should().BeTrue();
}

3. Cross-Platform Reproducibility (Tasks 15-18)

Tests that identical inputs produce identical outputs on different operating systems:

4. Golden Baseline Verification (Tasks 19-21)

Tests that current results match a pre-approved baseline:

// src/__Tests/__Benchmarks/determinism/golden-baseline/e2e-hashes.json
{
  "verdict_hash": "sha256:abc123...",
  "manifest_hash": "sha256:def456...",
  "envelope_hash": "sha256:ghi789...",
  "updated_at": "2025-06-15T12:00:00Z",
  "updated_by": "ci",
  "commit": "abc123def456"
}

Troubleshooting Reproducibility Failures

Common Causes

1. Non-Deterministic Ordering

Symptom: Different verdict hashes despite identical inputs.

Diagnosis:

// Check if collections are being ordered
var comparison = ManifestComparer.Compare(result1, result2);
var report = ManifestComparer.GenerateDiffReport(comparison);
Console.WriteLine(report);

Solution: Ensure all collections are sorted before hashing:

// Bad - non-deterministic
var findings = results.ToList();

// Good - deterministic
var findings = results.OrderBy(f => f.CveId, StringComparer.Ordinal)
                     .ThenBy(f => f.Purl, StringComparer.Ordinal)
                     .ToList();

2. Timestamp Drift

Symptom: Bundle manifests differ in createdAt field.

Diagnosis:

var jsonComparison = ManifestComparer.CompareJson(
    result1.BundleManifest, 
    result2.BundleManifest);

Solution: Use frozen timestamps in tests:

// In test fixture
public DateTimeOffset FrozenTimestamp { get; } = 
    new DateTimeOffset(2025, 6, 15, 12, 0, 0, TimeSpan.Zero);

3. Platform-Specific Behavior

Symptom: Tests pass on Ubuntu but fail on Windows/macOS.

Common causes:

• Line ending differences (\n vs \r\n)
• Path separator differences (/ vs \)
• Unicode normalization differences
• Floating-point representation differences

Diagnosis:

# Download artifacts from all platforms
# Compare hex dumps
xxd ubuntu-manifest.bin > ubuntu.hex
xxd windows-manifest.bin > windows.hex
diff ubuntu.hex windows.hex

Solution: Use platform-agnostic serialization:

// Use canonical JSON
var json = CanonJson.Serialize(data);

// Normalize line endings
var normalized = content.Replace("\r\n", "\n");

4. Key/Signature Differences

Symptom: Envelope hashes differ despite identical payloads.

Diagnosis:

// Compare envelope structure
var envelope1 = JsonSerializer.Deserialize<DsseEnvelope>(result1.EnvelopeBytes);
var envelope2 = JsonSerializer.Deserialize<DsseEnvelope>(result2.EnvelopeBytes);

// Check if payloads match
envelope1.Payload.SequenceEqual(envelope2.Payload).Should().BeTrue();

Solution: Use deterministic key generation:

// Generate key from fixed seed for reproducibility
private static ECDsa GenerateDeterministicKey(int seed)
{
    var rng = new DeterministicRng(seed);
    var keyBytes = new byte[32];
    rng.GetBytes(keyBytes);
    // ... create key from bytes
}

Debugging Tools

ManifestComparer

// Full comparison
var comparison = ManifestComparer.Compare(expected, actual);

// Multiple results
var multiComparison = ManifestComparer.CompareMultiple(results);

// Detailed report
var report = ManifestComparer.GenerateDiffReport(comparison);

// Hex dump for byte-level debugging
var hexDump = ManifestComparer.GenerateHexDump(expected.BundleManifest, actual.BundleManifest);

JSON Comparison

var jsonComparison = ManifestComparer.CompareJson(
    expected.BundleManifest,
    actual.BundleManifest);

foreach (var diff in jsonComparison.Differences)
{
    Console.WriteLine($"Path: {diff.Path}");
    Console.WriteLine($"Expected: {diff.Expected}");
    Console.WriteLine($"Actual: {diff.Actual}");
}

Updating the Golden Baseline

When intentional changes affect reproducibility (e.g., new fields, algorithm changes):

1. Manual Update

# Run tests and capture new hashes
dotnet test src/__Tests/Integration/StellaOps.Integration.E2E/ \
  --results-directory ./TestResults

# Update baseline (path must match the golden-baseline location below)
cp ./TestResults/verdict_hash.txt src/__Tests/__Benchmarks/determinism/golden-baseline/
# ... update e2e-hashes.json

2. CI Update (Recommended)

# Trigger workflow with update flag
# Via Gitea UI: Actions → E2E Reproducibility → Run workflow
# Set update_baseline = true

3. Approval Process

1. Create PR with baseline update
2. Explain why the change is intentional
3. Verify all platforms produce consistent results
4. Get approval from Platform Guild lead
5. Merge after CI passes

CI Workflow Reference

Jobs

Artifacts

Related Documentation

• Reproducibility Architecture
• VerdictId Content-Addressing
• DSSE Envelope Format
• Determinism Gates
• Determinism Verification

Sprint History

• 8200.0001.0004 - Initial E2E reproducibility test implementation
• 8200.0001.0001 - VerdictId content-addressing (dependency)
• 8200.0001.0002 - DSSE round-trip testing (dependency)