StellaOps.TestKit Usage Guide

Audience: Stella Ops developers writing unit, integration, and contract tests Source of truth: src/__Libraries/StellaOps.TestKit/ (verify API signatures against the code; this guide is reconciled against it but the code wins on disagreement) Target framework: net10.0 Test stack: xUnit v3 (xunit.v3.*), AwesomeAssertions, FsCheck (property tests), Testcontainers (PostgreSQL/Valkey), OpenTelemetry, ASP.NET WebApplicationFactory

Note: StellaOps.TestKit.csproj references xunit.v3.* and AwesomeAssertions (not the classic xUnit v2 or FluentAssertions). Consuming test projects should align with the same stack.

Overview

StellaOps.TestKit provides deterministic testing infrastructure for Stella Ops modules. It eliminates flaky tests, provides reproducible test primitives, and standardizes fixtures for integration testing.

Key Features

Installation

Add StellaOps.TestKit as a project reference to your test project. The library lives at src/__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj; adjust the relative path for your test project’s depth:

<ItemGroup>
  <ProjectReference Include="../../../__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj" />
</ItemGroup>

Quick Start Examples

1. Deterministic Time

Eliminate flaky tests caused by time-dependent logic:

using StellaOps.TestKit.Deterministic;
using Xunit;

[Fact]
public void Test_ExpirationLogic()
{
    // Arrange: Fix time at a known UTC timestamp
    using var time = new DeterministicTime(new DateTime(2026, 1, 15, 10, 30, 0, DateTimeKind.Utc));

    var expiresAt = time.UtcNow.AddHours(24);

    // Act: Advance time to just before expiration
    time.Advance(TimeSpan.FromHours(23));
    Assert.False(time.UtcNow > expiresAt);

    // Advance past expiration
    time.Advance(TimeSpan.FromHours(2));
    Assert.True(time.UtcNow > expiresAt);
}

API Reference: (StellaOps.TestKit.Deterministic.DeterministicTime, IDisposable)

2. Deterministic Random

Reproducible random sequences for property tests and fuzzing:

using StellaOps.TestKit.Deterministic;

[Fact]
public void Test_RandomIdGeneration()
{
    // Arrange: Same seed produces same sequence
    var random1 = new DeterministicRandom(seed: 42);
    var random2 = new DeterministicRandom(seed: 42);

    // Act
    var guid1 = random1.NextGuid();
    var guid2 = random2.NextGuid();

    // Assert: Reproducible GUIDs
    Assert.Equal(guid1, guid2);
}

[Fact]
public void Test_Shuffling()
{
    var random = new DeterministicRandom(seed: 100);
    var array = new[] { 1, 2, 3, 4, 5 };

    random.Shuffle(array);

    // Deterministic shuffle order
    Assert.NotEqual(new[] { 1, 2, 3, 4, 5 }, array);
}

API Reference: (StellaOps.TestKit.Deterministic.DeterministicRandom)

3. Canonical JSON Assertions

Verify JSON determinism for SBOM, VEX, and attestation outputs:

using StellaOps.TestKit.Assertions;

[Fact]
public void Test_SbomDeterminism()
{
    var sbom = new
    {
        SpdxVersion = "SPDX-3.0.1",
        Name = "MySbom",
        Packages = new[] { new { Name = "Pkg1", Version = "1.0" } }
    };

    // Verify deterministic serialization
    CanonicalJsonAssert.IsDeterministic(sbom, iterations: 100);

    // Verify expected hash (golden master)
    var expectedHash = "abc123..."; // Precomputed SHA-256
    CanonicalJsonAssert.HasExpectedHash(sbom, expectedHash);
}

[Fact]
public void Test_JsonPropertyExists()
{
    var vex = new
    {
        Document = new { Id = "VEX-2026-001" }
    };

    // Object property verification (dot-delimited path).
    CanonicalJsonAssert.ContainsProperty(vex, "Document.Id", "VEX-2026-001");
}

Path limitation: ContainsProperty splits the propertyPath on . and walks object properties only (case-insensitively). It does not support array indexing — a path like Statements[0].Vulnerability will not resolve, because Statements[0] is treated as a literal object-property name. To assert into arrays, serialize and inspect the element directly, or compare the whole structure with AreCanonicallyEqual / a snapshot.

API Reference: (StellaOps.TestKit.Assertions.CanonicalJsonAssert, static; hashing delegates to StellaOps.Canonical.Json.CanonJson)

4. Snapshot Testing

Golden master regression testing for complex outputs:

using StellaOps.TestKit.Assertions;

[Fact, Trait("Category", TestCategories.Snapshot)]
public void Test_SbomGeneration()
{
    var sbom = GenerateSbom(); // Your SBOM generation logic

    // Snapshot will be stored in Snapshots/TestSbomGeneration.json
    SnapshotAssert.MatchesSnapshot(sbom, "TestSbomGeneration");
}

// Update snapshots when intentional changes occur:
// UPDATE_SNAPSHOTS=1 dotnet test

Text and Binary Snapshots:

[Fact]
public void Test_LicenseText()
{
    var licenseText = GenerateLicenseNotice();
    SnapshotAssert.MatchesTextSnapshot(licenseText, "LicenseNotice");
}

[Fact]
public void Test_SignatureBytes()
{
    var signature = SignDocument(document);
    SnapshotAssert.MatchesBinarySnapshot(signature, "DocumentSignature");
}

API Reference:

5. PostgreSQL Fixture

Testcontainers-based PostgreSQL (postgres:16-alpine) for integration tests. PostgresFixture is an IAsyncLifetime fixture (it starts the container in InitializeAsync), usable as an xUnit IClassFixture<> or — to share one container across classes — via the supplied [Collection("Postgres")] collection fixture (PostgresCollection).

using StellaOps.TestKit.Fixtures;
using Npgsql;
using Xunit;

public class DatabaseTests : IClassFixture<PostgresFixture>
{
    private readonly PostgresFixture _fixture;

    public DatabaseTests(PostgresFixture fixture)
    {
        _fixture = fixture;
        // Optional: register migration scripts to apply per isolated session.
        // _fixture.RegisterMigrations("MyModule", "Migrations/001_init.sql");
    }

    [Fact, Trait("Category", TestCategories.Integration)]
    public async Task Test_DatabaseOperations()
    {
        // Get an isolated session (default isolation = SchemaPerTest).
        // Registered migrations are applied to the new schema automatically.
        await using var session = await _fixture.CreateSessionAsync(nameof(Test_DatabaseOperations));

        await using var connection = new NpgsqlConnection(session.ConnectionString);
        await connection.OpenAsync();

        await using var cmd = new NpgsqlCommand("SELECT version()", connection);
        var version = (string?)await cmd.ExecuteScalarAsync();
        Assert.NotNull(version);
    }
}

Migration helpers (there is no RunMigrationsAsync(DbConnection) method):

API Reference: (StellaOps.TestKit.Fixtures.PostgresFixture, IAsyncLifetime)

6. Valkey Fixture

Redis-compatible caching for integration tests. ValkeyFixture is an IAsyncLifetime fixture. The preferred entry point is CreateSessionAsync, which returns an isolated ValkeyTestSession exposing a ready IDatabase:

using StellaOps.TestKit.Fixtures;

public class CacheTests : IClassFixture<ValkeyFixture>
{
    private readonly ValkeyFixture _fixture;

    public CacheTests(ValkeyFixture fixture) => _fixture = fixture;

    [Fact, Trait("Category", TestCategories.Integration)]
    public async Task Test_CachingLogic()
    {
        await using var session = await _fixture.CreateSessionAsync(nameof(Test_CachingLogic));

        await session.Database.StringSetAsync("key", "value");
        var result = await session.Database.StringGetAsync("key");

        Assert.Equal("value", result.ToString());
    }
}

You can also connect directly with _fixture.ConnectionString / _fixture.GetDatabase(index) if you manage isolation yourself.

API Reference: (StellaOps.TestKit.Fixtures.ValkeyFixture, IAsyncLifetime)

7. HTTP Fixture Server

In-memory API contract testing:

using StellaOps.TestKit.Fixtures;

public class ApiTests : IClassFixture<HttpFixtureServer<Program>>
{
    private readonly HttpClient _client;

    public ApiTests(HttpFixtureServer<Program> fixture)
    {
        _client = fixture.CreateClient();
    }

    [Fact, Trait("Category", TestCategories.Contract)]
    public async Task Test_HealthEndpoint()
    {
        var response = await _client.GetAsync("/health");
        response.EnsureSuccessStatusCode();

        var body = await response.Content.ReadAsStringAsync();
        Assert.Contains("healthy", body);
    }
}

HTTP Message Handler Stub (Hermetic Tests):

[Fact]
public async Task Test_ExternalApiCall()
{
    var handler = new HttpMessageHandlerStub()
        .WhenRequest("https://api.example.com/data", HttpStatusCode.OK, "{\"status\":\"ok\"}");

    var httpClient = new HttpClient(handler);
    var response = await httpClient.GetAsync("https://api.example.com/data");

    Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}

API Reference: (StellaOps.TestKit.Fixtures)

8. OpenTelemetry Capture

Trace and span assertion helpers:

using StellaOps.TestKit.Observability;

[Fact]
public async Task Test_TracingBehavior()
{
    using var capture = new OtelCapture();

    // Execute code that emits traces
    await MyService.DoWorkAsync();

    // Assert traces
    capture.AssertHasSpan("MyService.DoWork");
    capture.AssertHasTag("user_id", "123");
    capture.AssertSpanCount(expectedCount: 3);

    // Verify parent-child hierarchy
    capture.AssertHierarchy("ParentSpan", "ChildSpan");
}

API Reference: (StellaOps.TestKit.Observability.OtelCapture, IDisposable)

9. Observability Contract Testing

Contract assertions for treating logs, metrics, and traces as APIs:

OTel Contract Testing:

using StellaOps.TestKit.Observability;

[Fact, Trait("Category", TestCategories.Contract)]
public async Task Test_SpanContracts()
{
    using var capture = new OtelCapture("MyService");

    await service.ProcessRequestAsync();

    // Verify required spans are present
    OTelContractAssert.HasRequiredSpans(capture, "ProcessRequest", "ValidateInput", "SaveResult");

    // Verify span attributes
    var span = capture.CapturedActivities.First();
    OTelContractAssert.SpanHasAttributes(span, "user_id", "tenant_id", "correlation_id");

    // Check attribute cardinality (prevent metric explosion)
    OTelContractAssert.AttributeCardinality(capture, "http_method", maxCardinality: 10);

    // Detect high-cardinality attributes globally
    OTelContractAssert.NoHighCardinalityAttributes(capture, threshold: 100);
}

Log Contract Testing:

using StellaOps.TestKit.Observability;
using System.Text.RegularExpressions;

[Fact]
public async Task Test_LogContracts()
{
    var logCapture = new List<CapturedLogRecord>();
    // ... capture logs during test execution ...

    // Verify required fields
    LogContractAssert.HasRequiredFields(logCapture[0], "CorrelationId", "TenantId");

    // Ensure no PII leakage
    var piiPatterns = new[]
    {
        new Regex(@"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b"),  // Email
        new Regex(@"\b\d{3}-\d{2}-\d{4}\b"),  // SSN
    };
    LogContractAssert.NoSensitiveData(logCapture, piiPatterns);

    // Verify log level appropriateness
    LogContractAssert.LogLevelAppropriate(logCapture[0], LogLevel.Information, LogLevel.Warning);

    // Ensure error logs have correlation for troubleshooting
    LogContractAssert.ErrorLogsHaveCorrelation(logCapture, "CorrelationId", "RequestId");
}

Metrics Contract Testing:

using StellaOps.TestKit.Observability;

[Fact]
public async Task Test_MetricsContracts()
{
    using var capture = new MetricsCapture("MyService");

    await service.ProcessMultipleRequests();

    // Verify required metrics exist
    MetricsContractAssert.HasRequiredMetrics(capture, "requests_total", "request_duration_seconds");

    // Check label cardinality bounds
    MetricsContractAssert.LabelCardinalityBounded(capture, "http_requests_total", maxLabels: 50);

    // Verify counter monotonicity
    MetricsContractAssert.CounterMonotonic(capture, "processed_items_total");

    // Verify gauge bounds
    MetricsContractAssert.GaugeInBounds(capture, "active_connections", minValue: 0, maxValue: 1000);
}

API Reference: (StellaOps.TestKit.Observability; all contract violations throw ContractViolationException)

OTelContractAssert (static):

LogContractAssert (static; operates over IEnumerable<CapturedLogRecord> / a single CapturedLogRecord):

MetricsContractAssert (static):

10. Evidence Chain Traceability

Link tests to requirements for regulatory compliance and audit trails:

Requirement Attribute:

using StellaOps.TestKit.Evidence;

[Fact]
[Requirement("REQ-AUTH-001", SprintTaskId = "AUTH-0127-001")]
public async Task Test_UserAuthentication()
{
    // Verify authentication works as required
}

[Fact]
[Requirement("REQ-AUDIT-002", SprintTaskId = "AUDIT-0127-003", ComplianceControl = "SOC2-AU-12")]
public void Test_AuditLogImmutability()
{
    // Verify audit logs cannot be modified
}

Filtering tests by requirement:

# Run tests for a specific requirement
dotnet test --filter "Requirement=REQ-AUTH-001"

# Run tests for a sprint task
dotnet test --filter "SprintTask=AUTH-0127-001"

# Run tests for a compliance control
dotnet test --filter "ComplianceControl=SOC2-AU-12"

Evidence Chain Assertions:

using StellaOps.TestKit.Evidence;

[Fact]
[Requirement("REQ-EVIDENCE-001")]
public void Test_ArtifactHashStability()
{
    var artifact = GenerateEvidence(input);

    // Verify artifact produces expected hash (golden master)
    EvidenceChainAssert.ArtifactHashStable(artifact, "abc123...expected-sha256...");
}

[Fact]
[Requirement("REQ-DETERMINISM-001")]
public void Test_EvidenceImmutability()
{
    // Verify generator produces identical output across iterations
    EvidenceChainAssert.ArtifactImmutable(() => GenerateEvidence(fixedInput), iterations: 100);
}

[Fact]
[Requirement("REQ-TRACE-001")]
public void Test_TraceabilityComplete()
{
    var requirementId = "REQ-EVIDENCE-001";
    var testId = "MyTests.TestMethod";
    var artifactHash = EvidenceChainAssert.ComputeSha256(artifact);

    // Verify all traceability components present
    EvidenceChainAssert.TraceabilityComplete(requirementId, testId, artifactHash);
}

Traceability Report Generation:

using StellaOps.TestKit.Evidence;

// Generate traceability matrix from test assemblies
var reporter = new EvidenceChainReporter();
reporter.AddAssembly(typeof(MyTests).Assembly);
var report = reporter.GenerateReport();

// Output as Markdown
Console.WriteLine(report.ToMarkdown());

// Output as JSON
Console.WriteLine(report.ToJson());

API Reference: (StellaOps.TestKit.Evidence)

11. Test Categories

Standardized trait constants for CI lane filtering:

using StellaOps.TestKit;

[Fact, Trait("Category", TestCategories.Unit)]
public void FastUnitTest() { }

[Fact, Trait("Category", TestCategories.Integration)]
public async Task SlowIntegrationTest() { }

[Fact, Trait("Category", TestCategories.Live)]
public async Task RequiresExternalServices() { }

CI Lane Filtering:

# Run only unit tests (fast, no dependencies)
dotnet test --filter "Category=Unit"

# Run all tests except Live
dotnet test --filter "Category!=Live"

# Run Integration + Contract tests
dotnet test --filter "Category=Integration|Category=Contract"

Available Categories (constants on StellaOps.TestKit.TestCategories; values equal their name, e.g. TestCategories.Unit == "Unit"):

Core lanes:

Pipeline-aligned lanes:

Storage-specific:

Schema evolution:

Distributed-systems / advanced:

Testing-enhancements lanes:

Blast-radius annotations (nested class TestCategories.BlastRadius, used as a separate BlastRadius trait, not the Category trait):

[Fact]
[Trait("Category", TestCategories.Integration)]
[Trait("BlastRadius", TestCategories.BlastRadius.Auth)]
public async Task TestTokenValidation() { }
// Filter: dotnet test --filter "BlastRadius=Auth|BlastRadius=Api"

12. Post-Incident Testing

Generate regression tests from production incidents:

Generate Test Scaffold from Incident:

using StellaOps.TestKit.Incident;

// Create incident metadata
var metadata = new IncidentMetadata
{
    IncidentId = "INC-2026-001",
    OccurredAt = DateTimeOffset.Parse("2026-01-15T10:30:00Z"),
    RootCause = "Race condition in concurrent bundle creation",
    AffectedModules = ["EvidenceLocker", "Policy"],
    Severity = IncidentSeverity.P1,
    Title = "Evidence bundle duplication"
};

// Generate test scaffold from replay manifest
var generator = new IncidentTestGenerator();
var scaffold = generator.GenerateFromManifestJson(manifestJson, metadata);

// Output generated test code
var code = scaffold.GenerateTestCode();
File.WriteAllText($"Tests/{scaffold.TestClassName}.cs", code);

Generated Test Structure:

[Trait("Category", TestCategories.PostIncident)]
[Trait("Incident", "INC-2026-001")]
[Trait("Severity", "P1")]
public sealed class Incident_INC_2026_001_Tests
{
    [Fact]
    public async Task Validates_RaceCondition_Fix()
    {
        // Arrange - fixtures from replay manifest
        // Act - execute the incident scenario
        // Assert - verify fix prevents recurrence
    }
}

Filter Post-Incident Tests:

# Run all post-incident tests
dotnet test --filter "Category=PostIncident"

# Run only P1/P2 tests (release-gating)
dotnet test --filter "Category=PostIncident&(Severity=P1|Severity=P2)"

# Run tests for a specific incident
dotnet test --filter "Incident=INC-2026-001"

API Reference: (StellaOps.TestKit.Incident)

See Post-Incident Testing Guide for complete documentation.

Additional Modules

The TestKit ships several further helpers beyond the sections above. These are part of StellaOps.TestKit today (verify signatures in source before relying on them); they are summarized here for completeness rather than fully documented:

Best Practices

1. Always Use TestCategories

Tag every test with the appropriate category:

[Fact, Trait("Category", TestCategories.Unit)]
public void MyUnitTest() { }

This enables CI lane filtering and improves test discoverability.

2. Prefer Deterministic Primitives

Avoid DateTime.UtcNow, Guid.NewGuid(), Random in tests. Use TestKit alternatives:

// ❌ Flaky test (time-dependent)
var expiration = DateTime.UtcNow.AddHours(1);

// ✅ Deterministic test
using var time = new DeterministicTime(DateTime.UtcNow);
var expiration = time.UtcNow.AddHours(1);

3. Use Snapshot Tests for Complex Outputs

For large JSON outputs (SBOM, VEX, attestations), snapshot testing is more maintainable than manual assertions:

// ❌ Brittle manual assertions
Assert.Equal("SPDX-3.0.1", sbom.SpdxVersion);
Assert.Equal(42, sbom.Packages.Count);
// ...hundreds of assertions...

// ✅ Snapshot testing
SnapshotAssert.MatchesSnapshot(sbom, "MySbomSnapshot");

4. Isolate Integration Tests

Use TestCategories to separate fast unit tests from slow integration tests:

[Fact, Trait("Category", TestCategories.Unit)]
public void FastTest() { /* no external dependencies */ }

[Fact, Trait("Category", TestCategories.Integration)]
public async Task SlowTest() { /* uses PostgresFixture */ }

In CI, run Unit tests first for fast feedback, then Integration tests in parallel.

5. Document Snapshot Baselines

When updating snapshots (UPDATE_SNAPSHOTS=1), add a commit message explaining why:

git commit -m "Update SBOM snapshot: added new package metadata fields"

This helps reviewers understand intentional vs. accidental changes.

Troubleshooting

Snapshot Mismatch

Error: Snapshot 'MySbomSnapshot' does not match expected.

Solution:

  1. Review diff manually (check Snapshots/MySbomSnapshot.json)
  2. If change is intentional: UPDATE_SNAPSHOTS=1 dotnet test
  3. Commit updated snapshot with explanation

Testcontainers Failure

Error: Docker daemon not running

Solution:

Determinism Failure

Error: CanonicalJsonAssert.IsDeterministic failed: byte arrays differ

Root Cause: Non-deterministic data in serialization (e.g., random GUIDs, timestamps)

Solution:

Migration Guide (Existing Tests)

Step 1: Add TestKit Reference

<ProjectReference Include="../../../__Libraries/StellaOps.TestKit/StellaOps.TestKit.csproj" />

Step 2: Replace Time-Dependent Code

Before:

var now = DateTime.UtcNow;

After:

using var time = new DeterministicTime(DateTime.UtcNow);
var now = time.UtcNow;

Step 3: Add Test Categories

[Fact] // Old
[Fact, Trait("Category", TestCategories.Unit)] // New

Step 4: Adopt Snapshot Testing (Optional)

For complex JSON assertions, replace manual checks with snapshots:

// Old
Assert.Equal(expected.SpdxVersion, actual.SpdxVersion);
// ...

// New
SnapshotAssert.MatchesSnapshot(actual, "TestName");

CI Integration

The CI lanes live under .gitea/workflows/ (e.g. dotnet-pr-tests.yml, test-manifest-execution.yml, test-architecture.yml, dotnet-nightly-sweep.yml). The repo does not ship a single test.yml; the snippet below is an illustrative example of how to filter by TestCategories in a workflow — adapt it to the existing lanes rather than adding a new file.

Example workflow (illustrative)

name: Test Suite
on: [push, pull_request]

jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '10.0.x'
      - name: Unit Tests (Fast)
        run: dotnet test --filter "Category=Unit" --logger "trx;LogFileName=unit-results.trx"
      - name: Upload Results
        uses: actions/upload-artifact@v4
        with:
          name: unit-test-results
          path: '**/unit-results.trx'

  integration:
    runs-on: ubuntu-latest
    services:
      docker:
        image: docker:dind
    steps:
      - uses: actions/checkout@v4
      - name: Integration Tests
        run: dotnet test --filter "Category=Integration" --logger "trx;LogFileName=integration-results.trx"

Support and Feedback

Changelog

This changelog records the early release history only. The current TestKit surface is substantially larger (observability/log/metrics contract assertions, evidence-chain traceability, post-incident generation, connector test bases, storage/cache/query templates, interop, longevity, environment-skew, blast-radius, and test-intent helpers). Treat src/__Libraries/StellaOps.TestKit/ as the authoritative inventory.

Initial release