Performance Baselines - Determinism Tests

Overview

This document tracks performance baselines for determinism tests. Baselines help detect performance regressions and ensure tests remain fast for rapid CI/CD feedback.

Last Updated: 2025-12-29 .NET Version: 10.0.100 Hardware Reference: GitHub Actions runners (windows-latest, ubuntu-latest, macos-latest)

Baseline Metrics

CGS (Canonical Graph Signature) Tests

File: src/__Tests/Determinism/CgsDeterminismTests.cs

Regression Threshold: If any test exceeds baseline by >2x, investigate.

SBOM Lineage Tests

File: src/SbomService/__Tests/StellaOps.SbomService.Lineage.Tests/LineageDeterminismTests.cs

VexLens Truth Table Tests

File: src/VexLens/__Tests/StellaOps.VexLens.Tests/Consensus/VexLensTruthTableTests.cs

Scheduler Resilience Tests

File: src/Scheduler/__Tests/StellaOps.Scheduler.Tests/

Note: Scheduler tests use Testcontainers (PostgreSQL), adding ~2s startup overhead.

Platform Comparison

Average Speed Factor (relative to Linux Ubuntu)

Alpine Performance: Alpine is consistently ~60% slower due to musl libc differences. This is expected and acceptable.

Historical Trends

2025-12-29 (Baseline Establishment)

• .NET Version: 10.0.100
• Total Tests: 79
• Total Execution Time: ~25 seconds (all platforms, sequential)
• Status: ✅ All tests passing

Key Metrics:

• CGS determinism tests: <3s per platform
• Lineage determinism tests: <3s per platform
• VexLens truth tables: <2s per platform
• Scheduler resilience: <30s per platform (includes Testcontainers overhead)

Regression Detection

Automated Monitoring

CI/CD workflow .gitea/workflows/cross-platform-determinism.yml tracks execution time and fails if:

- name: Check for performance regression
  run: |
    # Fail if CGS test suite exceeds 3 seconds on Linux
    if [ $CGS_SUITE_TIME_MS -gt 3000 ]; then
      echo "ERROR: CGS test suite exceeded 3s baseline (${CGS_SUITE_TIME_MS}ms)"
      exit 1
    fi

    # Fail if Alpine is >3x slower than Linux (expected is ~1.6x)
    ALPINE_FACTOR=$(echo "$ALPINE_TIME_MS / $LINUX_TIME_MS" | bc -l)
    if (( $(echo "$ALPINE_FACTOR > 3.0" | bc -l) )); then
      echo "ERROR: Alpine is >3x slower than Linux (factor: $ALPINE_FACTOR)"
      exit 1
    fi

Manual Benchmarking

Run benchmarks locally to compare before/after changes:

cd src/__Tests/Determinism

# Run with detailed timing
dotnet test --logger "console;verbosity=detailed" | tee benchmark-$(date +%Y%m%d).log

# Extract timing
grep -E "Test Name:|Duration:" benchmark-*.log

Example Output:

Test Name: CgsHash_WithKnownEvidence_MatchesGoldenHash
Duration: 87ms

Test Name: CgsHash_SameInput_ProducesIdenticalHash_Across10Iterations
Duration: 850ms

BenchmarkDotNet Integration (Future)

For precise micro-benchmarks:

// src/__Tests/__Benchmarks/CgsHashBenchmarks.cs
[MemoryDiagnoser]
[MarkdownExporter]
public class CgsHashBenchmarks
{
    [Benchmark]
    public string ComputeCgsHash_SmallEvidence()
    {
        var evidence = CreateSmallEvidencePack();
        var policyLock = CreatePolicyLock();
        var service = new VerdictBuilderService(NullLogger.Instance);
        return service.ComputeCgsHash(evidence, policyLock);
    }

    [Benchmark]
    public string ComputeCgsHash_LargeEvidence()
    {
        var evidence = CreateLargeEvidencePack();  // 100 VEX documents
        var policyLock = CreatePolicyLock();
        var service = new VerdictBuilderService(NullLogger.Instance);
        return service.ComputeCgsHash(evidence, policyLock);
    }
}

Run:

dotnet run -c Release --project src/__Tests/__Benchmarks/StellaOps.Benchmarks.Determinism.csproj

Optimization Strategies

Strategy 1: Reduce Allocations

Before:

for (int i = 0; i < 10; i++)
{
    var leaves = new List<string>();  // ❌ Allocates every iteration
    leaves.Add(ComputeHash(input));
}

After:

var leaves = new List<string>(capacity: 10);  // ✅ Pre-allocate
for (int i = 0; i < 10; i++)
{
    leaves.Clear();
    leaves.Add(ComputeHash(input));
}

Strategy 2: Use Spanfor Hashing

Before:

var bytes = Encoding.UTF8.GetBytes(input);  // ❌ Allocates byte array
var hash = SHA256.HashData(bytes);

After:

Span<byte> buffer = stackalloc byte[256];  // ✅ Stack allocation
var bytesWritten = Encoding.UTF8.GetBytes(input, buffer);
var hash = SHA256.HashData(buffer[..bytesWritten]);

Strategy 3: Cache Expensive Computations

Before:

[Fact]
public void Test()
{
    var service = CreateService();  // ❌ Recreates every test
    // ...
}

After:

private readonly MyService _service;  // ✅ Reuse across tests

public MyTests()
{
    _service = CreateService();
}

Strategy 4: Parallel Test Execution

xUnit runs tests in parallel by default. To disable for specific tests:

[Collection("Sequential")]  // Disable parallelism
public class MySlowTests
{
    // Tests run sequentially within this class
}

Performance Regression Examples

Example 1: Unexpected Allocations

Symptom: Test time increased from 85ms to 450ms after refactoring.

Cause: Accidental string concatenation in loop:

// Before: 85ms
var hash = string.Join("", hashes);

// After: 450ms (BUG!)
var result = "";
foreach (var h in hashes)
{
    result += h;  // ❌ Creates new string every iteration!
}

Fix: Use StringBuilder:

var sb = new StringBuilder();
foreach (var h in hashes)
{
    sb.Append(h);  // ✅ Efficient
}
var result = sb.ToString();

Example 2: Excessive I/O

Symptom: Test time increased from 100ms to 2,500ms.

Cause: Reading file from disk every iteration:

for (int i = 0; i < 10; i++)
{
    var data = File.ReadAllText("test-data.json");  // ❌ Disk I/O every iteration!
    ProcessData(data);
}

Fix: Read once, reuse:

var data = File.ReadAllText("test-data.json");  // ✅ Read once
for (int i = 0; i < 10; i++)
{
    ProcessData(data);
}

Example 3: Inefficient Sorting

Symptom: Test time increased from 165ms to 950ms after adding VEX documents.

Cause: Sorting inside loop:

for (int i = 0; i < 10; i++)
{
    var sortedVex = vexDocuments.OrderBy(v => v).ToList();  // ❌ Sorts every iteration!
    ProcessVex(sortedVex);
}

Fix: Sort once, reuse:

var sortedVex = vexDocuments.OrderBy(v => v).ToList();  // ✅ Sort once
for (int i = 0; i < 10; i++)
{
    ProcessVex(sortedVex);
}

Monitoring and Alerts

Slack Alerts

Configure alerts for performance regressions:

# .gitea/workflows/cross-platform-determinism.yml
- name: Notify on regression
  if: failure() && steps.performance-check.outcome == 'failure'
  uses: slackapi/slack-github-action@v1
  with:
    payload: |
      {
        "text": "⚠️ Performance regression detected in determinism tests",
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "*CGS Test Suite Exceeded Baseline*\n\nBaseline: 3s\nActual: ${{ steps.performance-check.outputs.duration }}s\n\nPlatform: Linux Ubuntu\nCommit: ${{ github.sha }}"
            }
          }
        ]
      }

Grafana Dashboard

Track execution time over time:

# Prometheus query
histogram_quantile(0.95,
  rate(determinism_test_duration_seconds_bucket{test="CgsHash_10Iterations"}[5m])
)

Dashboard Panels:

1. Test duration (p50, p95, p99) over time
2. Platform comparison (Windows vs Linux vs macOS vs Alpine)
3. Test failure rate by platform
4. Execution time distribution (histogram)

References

• CI/CD Workflow: .gitea/workflows/cross-platform-determinism.yml
• Test README: src/__Tests/Determinism/README.md
• Developer Guide: docs/technical/testing/DETERMINISM_DEVELOPER_GUIDE.md
• Batch Summary: docs/implplan/archived/2025-12-29-completed-sprints/BATCH_20251229_BE_COMPLETION_SUMMARY.md

Changelog

2025-12-29 - Initial Baselines

• Established baselines for CGS, Lineage, VexLens, and Scheduler tests
• Documented platform speed factors (Alpine 1.6x, macOS 1.1x, Windows 1.02x)
• Set regression thresholds (>2x baseline triggers investigation)
• Configured CI/CD performance monitoring