Test Strategy

This page describes how Stella Ops organizes its automated tests — the category model, how categories are discovered and traited, and how they map onto CI. It is written for engineers adding tests, and for anyone who needs to understand what gates a pull request versus what runs on a schedule.

Altitude note. Category timeouts, the discovery globs, and the CI job layout drift as the pipeline evolves; the authoritative definitions live in .gitea/scripts/test/ and the workflow files under .gitea/workflows/. The example workflow YAML below uses Gitea Actions (GitHub Actions–compatible) syntax and is illustrative — verify specifics against the workflow sources before relying on them. See the CI/CD overview.

Test Category Overview

Stella Ops uses a tiered testing strategy organized into 13 test categories: six that gate pull requests and seven extended categories that run on a schedule or on demand.

PR-Gating Tests (Required for Merge)

Extended Tests (Scheduled/On-Demand)

Test Discovery

Automatic Discovery

The test-matrix.yml workflow automatically discovers all test projects:

# Discovery pattern
find src \( \
  -name "*.Tests.csproj" \
  -o -name "*UnitTests.csproj" \
  -o -name "*SmokeTests.csproj" \
  -o -name "*FixtureTests.csproj" \
  -o -name "*IntegrationTests.csproj" \
\) -type f \
  ! -path "*/node_modules/*" \
  ! -path "*/bin/*" \
  ! -path "*/obj/*" \
  ! -name "StellaOps.TestKit.csproj" \
  ! -name "*Testing.csproj"

Test Category Trait

Tests are categorized using xUnit traits:

[Trait("Category", "Unit")]
public class MyUnitTests
{
    [Fact]
    public void Should_Do_Something()
    {
        // ...
    }
}

[Trait("Category", "Integration")]
public class MyIntegrationTests
{
    [Fact]
    public void Should_Connect_To_Database()
    {
        // ...
    }
}

Running Specific Categories

# Run Unit tests only
dotnet test --filter "Category=Unit"

# Run multiple categories
dotnet test --filter "Category=Unit|Category=Integration"

# Run excluding a category
dotnet test --filter "Category!=Performance"

Test Infrastructure

Shared Libraries

Testcontainers

Integration tests use Testcontainers for isolated dependencies:

public class PostgresFixture : IAsyncLifetime
{
    private readonly PostgreSqlContainer _container = new PostgreSqlBuilder()
        .WithImage("postgres:16")
        .WithDatabase("test_db")
        .Build();

    public string ConnectionString => _container.GetConnectionString();

    public Task InitializeAsync() => _container.StartAsync();
    public Task DisposeAsync() => _container.DisposeAsync().AsTask();
}

Ground Truth Corpus

Golden tests use a corpus of known-good outputs:

src/__Tests/__Datasets/
├── scanner/
│   ├── golden/
│   │   ├── npm-package.expected.json
│   │   ├── dotnet-project.expected.json
│   │   └── container-image.expected.json
│   └── fixtures/
│       ├── npm-package/
│       ├── dotnet-project/
│       └── container-image/
└── concelier/
    ├── golden/
    └── fixtures/

CI/CD Integration

Test Matrix Workflow

# Simplified test-matrix.yml structure
jobs:
  discover:
    # Find all test projects
    outputs:
      test-projects: ${{ steps.find.outputs.projects }}

  pr-gating-tests:
    strategy:
      matrix:
        include:
          - category: Unit
            timeout: 20
          - category: Architecture
            timeout: 15
          - category: Contract
            timeout: 15
          - category: Security
            timeout: 25
          - category: Golden
            timeout: 25
    steps:
      - run: .gitea/scripts/test/run-test-category.sh "${{ matrix.category }}"

  integration:
    services:
      postgres:
        image: postgres:16
    steps:
      - run: .gitea/scripts/test/run-test-category.sh Integration

  summary:
    needs: [discover, pr-gating-tests, integration]

Test Results

All tests produce TRX (Visual Studio Test Results) files:

# Output structure
TestResults/
├── Unit/
│   ├── src_Scanner___Tests_StellaOps.Scanner.Tests-unit.trx
│   └── src_Authority___Tests_StellaOps.Authority.Tests-unit.trx
├── Integration/
│   └── ...
└── Combined/
    └── test-results-combined.trx

Coverage Collection

# Collect coverage for Unit tests
- run: |
    .gitea/scripts/test/run-test-category.sh Unit --collect-coverage

Coverage reports are generated in Cobertura format and converted to HTML.

Test Categories Deep Dive

Unit Tests

Purpose: Test isolated components without external dependencies.

Characteristics:

• No I/O (database, network, file system)
• No async waits or delays
• Fast execution (< 100ms per test)
• High parallelism

Example:

[Trait("Category", "Unit")]
public class VexPolicyBinderTests
{
    [Fact]
    public void Bind_WithValidPolicy_ReturnsSuccess()
    {
        var binder = new VexPolicyBinder();
        var policy = new VexPolicy { /* ... */ };

        var result = binder.Bind(policy);

        Assert.True(result.IsSuccess);
    }
}

Architecture Tests

Purpose: Enforce architectural rules and dependency constraints.

Rules Enforced:

• Layer dependencies (UI → Application → Domain → Infrastructure)
• Namespace conventions
• Circular dependency prevention
• Interface segregation

Example:

[Trait("Category", "Architecture")]
public class DependencyTests
{
    [Fact]
    public void Domain_Should_Not_Depend_On_Infrastructure()
    {
        var result = Types.InAssembly(typeof(DomainMarker).Assembly)
            .That().ResideInNamespace("StellaOps.Domain")
            .ShouldNot().HaveDependencyOn("StellaOps.Infrastructure")
            .GetResult();

        Assert.True(result.IsSuccessful);
    }
}

Contract Tests

Purpose: Validate API contracts are maintained.

Checks:

• Request/response schemas
• OpenAPI specification compliance
• Backward compatibility

Example:

[Trait("Category", "Contract")]
public class VulnerabilityApiContractTests
{
    [Fact]
    public async Task GetVulnerability_ReturnsExpectedSchema()
    {
        var response = await _client.GetAsync("/api/v1/vulnerabilities/CVE-2024-1234");

        await Verify(response)
            .UseDirectory("Snapshots")
            .UseMethodName("GetVulnerability");
    }
}

Integration Tests

Purpose: Test component integration with real dependencies.

Dependencies:

• PostgreSQL (via Testcontainers)
• Valkey/Redis (via Testcontainers)
• File system

Example:

[Trait("Category", "Integration")]
public class VulnerabilityRepositoryTests : IClassFixture<PostgresFixture>
{
    private readonly PostgresFixture _fixture;

    [Fact]
    public async Task Save_AndRetrieve_Vulnerability()
    {
        var repo = new VulnerabilityRepository(_fixture.ConnectionString);
        var vuln = new Vulnerability { Id = "CVE-2024-1234" };

        await repo.SaveAsync(vuln);
        var retrieved = await repo.GetAsync("CVE-2024-1234");

        Assert.Equal(vuln.Id, retrieved.Id);
    }
}

Security Tests

Purpose: Validate security controls and assertions.

Checks:

• Input validation
• Authorization enforcement
• Cryptographic operations
• Secrets handling

Example:

[Trait("Category", "Security")]
public class AuthorizationTests
{
    [Fact]
    public async Task Unauthorized_User_Cannot_Access_Admin_Endpoint()
    {
        var client = _factory.CreateClient();
        client.DefaultRequestHeaders.Authorization = null;

        var response = await client.GetAsync("/api/admin/settings");

        Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
    }
}

Golden Tests

Purpose: Verify output matches known-good corpus.

Example:

[Trait("Category", "Golden")]
public class ScannerGoldenTests
{
    [Theory]
    [InlineData("npm-package")]
    [InlineData("dotnet-project")]
    public async Task Scan_MatchesGoldenOutput(string fixture)
    {
        var scanner = new ContainerScanner();
        var result = await scanner.ScanAsync($"fixtures/{fixture}");

        await Verify(result)
            .UseDirectory("golden")
            .UseFileName(fixture);
    }
}

Performance Testing

BenchmarkDotNet

[Trait("Category", "Benchmark")]
[MemoryDiagnoser]
public class ScannerBenchmarks
{
    [Benchmark]
    public async Task ScanSmallImage()
    {
        await _scanner.ScanAsync(_smallImage);
    }

    [Benchmark]
    public async Task ScanLargeImage()
    {
        await _scanner.ScanAsync(_largeImage);
    }
}

Performance SLOs

Troubleshooting

Tests Fail in CI but Pass Locally

1. Check timezone - CI uses TZ=UTC
2. Check parallelism - CI runs tests in parallel
3. Check container availability - Testcontainers requires Docker
4. Check file paths - Case sensitivity on Linux

Flaky Tests

1. Add retry logic for network operations
2. Use proper async/await - no Task.Run for async operations
3. Isolate shared state - use fresh fixtures per test
4. Increase timeouts for integration tests

Missing Test Category

Ensure your test class has the correct trait:

[Trait("Category", "Unit")]  // Add this!
public class MyTests
{
    // ...
}

Related documentation

• CI/CD Overview
• Workflow Triggers
• CI Quality Gates
• Test Trait Taxonomy — canonical category/trait catalog
• Test Manifest — machine-readable suite catalog
• Test Suite Overview