Connector Fixture Discipline

Audience: engineers building or maintaining Stella Ops advisory (Concelier) and VEX (Excititor) connectors, plus reviewers checking that a new connector carries the required test layers.

This document defines the testing discipline for Stella Ops Concelier and Excititor connectors. All connectors must follow these patterns to ensure consistent, deterministic, and offline-capable testing.

Overview

Connector tests follow Model C1 (Connector/External) from the testing strategy:

  1. Fixture-based parser tests — Raw upstream payload → normalized internal model (offline)
  2. Resilience tests — Partial/bad input → deterministic failure classification
  3. Security tests — URL allowlist, redirect handling, payload limits
  4. Live smoke tests — Schema drift detection (opt-in, non-gating)

The shared base classes and helpers live in StellaOps.TestKit.Connectors (src/__Libraries/StellaOps.TestKit/Connectors/). The base classes referenced below are the actual types in that namespace:

ConcernBase classNotes
Parser snapshotConnectorParserTestBase<TRaw, TNormalized>Abstract DeserializeRaw/Parse/DeserializeNormalized; VerifyParseSnapshot/VerifyDeterministicParse helpers
Fetch + parseConnectorFetchTestBase<TConnector, TNormalized>Wires ConnectorHttpFixture for canned responses
ResilienceConnectorResilienceTestBaseNon-generic; abstract TryParse/TryFetchAsync return (bool Success, string? ErrorCategory)
SecurityConnectorSecurityTestBaseNon-generic; abstract ConnectorName/AllowedUrlPatterns/TryFetchUrlAsync; CreateGzipBomb/CreateNestedGzipBomb helpers
Live schema driftConnectorLiveSchemaTestBaseAbstract FixturesDirectory/ConnectorName/GetTestCases; gated by STELLAOPS_LIVE_TESTS=true

Note: connectors are not required to inherit these bases. Several (e.g. the NVD resilience tests and the GHSA security tests) test the connector’s actual mapper or the live ConnectorTestHarness directly with plain assertions, while still carrying the lane/category traits described below. The bases standardize the common patterns but do not gate.

1. Directory Structure

Each connector test project follows this structure:

src/<Module>/__Tests/StellaOps.<Module>.Connector.<Source>.Tests/
├── StellaOps.<Module>.Connector.<Source>.Tests.csproj
├── <Source>/
│   ├── Fixtures/                      # Raw upstream payloads (copied to output)
│   │   ├── <source>-typical.json      # Typical advisory payload
│   │   ├── <source>-edge-<case>.json  # Edge case payloads
│   │   └── <source>-invalid-...json   # Malformed/invalid payloads
│   ├── <Source>ParserSnapshotTests.cs # Parser snapshot + determinism tests
│   └── <Source>ResilienceTests.cs     # Resilience tests
└── Expected/
    └── <source>-<id>.canonical.json   # Canonical JSON snapshots

The fixture/snapshot file names above are conventions, not enforced rules — actual projects vary (e.g. NVD uses nvd-window-N.json, nvd-invalid-schema.json, <id>.canonical.v2.json). What is load-bearing is that FixturesDirectory and ExpectedDirectory resolve under the test assembly’s AppContext.BaseDirectory, so the fixture/expected files must be copied to output (<CopyToOutputDirectory> / Content/None items in the .csproj).

Example: NVD Connector (actual on-disk layout)

Note that the connector subfolder owns its own Fixtures/ directory, while expected canonical snapshots live in a sibling top-level Expected/ directory. This is the layout NvdParserSnapshotTests resolves via FixturesDirectory => Nvd/Fixtures and ExpectedDirectory => Expected.

src/Concelier/__Tests/StellaOps.Concelier.Connector.Nvd.Tests/
├── Nvd/
│   ├── Fixtures/
│   │   ├── nvd-window-1.json
│   │   ├── nvd-window-2.json
│   │   ├── nvd-multipage-1.json
│   │   ├── nvd-multipage-2.json
│   │   ├── nvd-invalid-schema.json
│   │   └── conflict-nvd.json
│   ├── NvdParserSnapshotTests.cs
│   └── NvdResilienceTests.cs
└── Expected/
    ├── nvd-window-1-CVE-2024-0001.canonical.v2.json
    └── conflict-nvd.canonical.v2.json

2. Fixture-Based Parser Tests

Purpose

Test that the parser correctly transforms raw upstream payloads into normalized internal models without network access.

Pattern

ConnectorParserTestBase<TRaw, TNormalized> is abstract: a subclass supplies FixturesDirectory, the (de)serialization hooks (DeserializeRaw, Parse, DeserializeNormalized), and an optional SerializeToCanonical override, then calls the VerifyParseSnapshot / VerifyDeterministicParse helpers from [Fact] methods. The base reads fixtures with ReadFixture(name) and expected snapshots with ReadExpected(name) — there is no LoadFixture<T>() / AssertMatchesSnapshot() / Parser.Parse() surface.

using System.Text.Json;
using StellaOps.Concelier.Models;
using StellaOps.TestKit.Connectors;
using Xunit;

public sealed class NvdParserSnapshotTests
    : ConnectorParserTestBase<JsonDocument, IReadOnlyList<Advisory>>
{
    private static readonly string BaseDirectory = AppContext.BaseDirectory;

    protected override string FixturesDirectory =>
        Path.Combine(BaseDirectory, "Nvd", "Fixtures");

    protected override string ExpectedDirectory =>
        Path.Combine(BaseDirectory, "Expected");

    protected override JsonDocument DeserializeRaw(string json) =>
        JsonDocument.Parse(json);

    protected override IReadOnlyList<Advisory> Parse(JsonDocument raw) =>
        NvdMapper.Map(raw, CreateTestDocumentRecord(), recordedAt: FixedTime);

    protected override IReadOnlyList<Advisory> DeserializeNormalized(string json) =>
        JsonSerializer.Deserialize<List<Advisory>>(json) ?? new();

    [Fact]
    [Trait("Lane", "Unit")]
    [Trait("Category", "Snapshot")]
    public void ParseNvdWindow1_CVE20240001_ProducesExpectedOutput()
    {
        VerifyParseSnapshot("nvd-window-1.json", "nvd-window-1-CVE-2024-0001.canonical.v2.json");
    }

    [Fact]
    [Trait("Lane", "Unit")]
    [Trait("Category", "Determinism")]
    public void ParseNvdWindow1_IsDeterministic()
    {
        VerifyDeterministicParse("nvd-window-1.json");
    }
}

Tests carry both a Lane trait (Unit/Security/Integration/Live, consumed by the CI lane filters — see CI Lane Filters) and a Category trait drawn from StellaOps.TestKit.TestCategories (e.g. Snapshot, Determinism, Resilience, Security, Live). The actual NVD tests apply both.

Fixture Requirements

Recommended naming conventions (actual projects vary — see the note in §1):

TypeNaming ConventionPurpose
Typical<source>-typical.json (NVD: nvd-window-N.json)Normal advisory with all common fields
Edge case<source>-edge-<case>.json (NVD: nvd-multipage-N.json)Unusual but valid payloads
Error<source>-error-<type>.json (NVD: nvd-invalid-schema.json)Malformed/invalid payloads
Expected<id>.canonical.json under Expected/Expected normalized output
Canonical<source>-<id>.canonical.v2.jsonDeterministic JSON snapshot

Minimum Coverage

Each connector must have fixtures for:

3. Resilience Tests

Purpose

Verify that connectors handle malformed input gracefully with deterministic failure classification.

Pattern

ConnectorResilienceTestBase is non-generic. A subclass implements two abstract hooks that classify outcomes deterministically:

The base then contributes ready-made [Fact]s that assert the error category is deterministic across repeated runs (MissingRequiredFields_…, MalformedJson_…, EmptyInput_…, NullInput_…, HttpError_…, HttpNotFound_…, Timeout_…). There is no ConnectorErrorKind enum and no result.Error / result.Warnings surface — failure classification is a free-form string? ErrorCategory and the invariant under test is determinism, not a fixed taxonomy.

public sealed class NvdResilienceTests : ConnectorResilienceTestBase
{
    protected override string FixturesDirectory =>
        Path.Combine(AppContext.BaseDirectory, "Nvd", "Fixtures");

    protected override (bool Success, string? ErrorCategory) TryParse(string json)
    {
        try
        {
            using var doc = JsonDocument.Parse(json);
            var advisories = NvdMapper.Map(doc, CreateTestDocumentRecord(), FixedTime);
            return (true, null);
        }
        catch (JsonException)
        {
            return (false, "malformed-json");
        }
    }

    protected override Task<(bool, string?)> TryFetchAsync(string url, CancellationToken ct = default)
        => /* drive the connector via HttpFixture and classify the failure */;
}

Connectors may also test their mapper directly without inheriting the base — the actual NvdResilienceTests does exactly this, asserting that missing/empty vulnerabilities arrays return an empty list, entries without a CVE object are skipped, invalid dates parse to null, and that error handling is deterministic across three runs. Both styles carry [Trait("Lane", "Unit")] + [Trait("Category", "Resilience")].

Required Test Cases

Express the expected behaviour as a deterministic (Success, ErrorCategory) outcome or a deterministic mapper result; the base’s invariant is that the same input always yields the same classification.

CaseExpected BehaviorTraits
Missing required fieldSkip/empty result, deterministicLane=Unit, Category=Resilience
Invalid date formatParsed with null date, deterministicLane=Unit, Category=Resilience
Malformed JSONSuccess=false, deterministic ErrorCategoryLane=Unit, Category=Resilience
Empty / null inputSuccess=false, deterministic ErrorCategoryLane=Unit, Category=Resilience
Unknown enum valueParses without throwing, deterministicLane=Unit, Category=Resilience
Truncated payloadJsonException on parse, deterministicLane=Unit, Category=Resilience

4. Security Tests

Purpose

Verify that connectors enforce security boundaries for network operations.

Pattern

ConnectorSecurityTestBase is non-generic. A subclass supplies ConnectorName, the AllowedUrlPatterns list (patterns support *, *.suffix, and prefix*), and the abstract TryFetchUrlAsync(url) hook returning (bool Allowed, string? ErrorMessage). It also exposes the virtual knobs MaxPayloadSizeBytes (default 50 MB) and MaxDecompressionRatio (default 100:1), the IsUrlInAllowlist(url) helper, and the static CreateGzipBomb / CreateNestedGzipBomb payload builders. The base ships ready-made [Theory]/[Fact]s — NonAllowlistedUrl_IsRejected, HttpsToHttpDowngrade_IsRejected, InternalUrl_IsRejected (SSRF: localhost, link-local, AWS/GCP metadata), MaxRedirectChain_IsEnforced. There is no SecurityException, PayloadTooLargeException, or CreateConnector(allowedHosts: …) / connector.FetchAsync(url) surface; rejection is expressed as Allowed == false.

public sealed class NvdSecurityTests : ConnectorSecurityTestBase
{
    protected override string ConnectorName => "NVD";

    protected override IReadOnlyList<string> AllowedUrlPatterns => new[]
    {
        "services.nvd.nist.gov",
    };

    protected override Task<(bool Allowed, string? ErrorMessage)> TryFetchUrlAsync(
        string url, CancellationToken ct = default)
    {
        // Drive the connector's actual URL validation; return Allowed=false on reject.
        var allowed = IsUrlInAllowlist(url);
        return Task.FromResult((allowed, allowed ? null : "url not in allowlist"));
    }
}

In practice some connectors (e.g. GHSA) skip the base and assert security properties against the live ConnectorTestHarness instead — verifying that external reference URLs are never followed (no SSRF), that oversized payloads do not OOM, that gzip/nested-gzip bombs are bounded (via CreateGzipBomb/CreateNestedGzipBomb), that malicious IDs are sanitized, and that a 429 response is not retried aggressively. These tests carry [Trait("Category", TestCategories.Unit)] + [Trait("Category", TestCategories.Security)].

Required Security Tests

TestPurposeTrait
URL allowlistBlock requests to unauthorized hostsCategory=Security
HTTPS→HTTP downgradeReject insecure schemeCategory=Security
Internal/SSRF URLsBlock localhost, link-local, cloud metadataCategory=Security
Max payload sizeReject/bound oversized responsesCategory=Security
Decompression bombReject/bound gzip and nested-gzip bombsCategory=Security
Rate limitingFail fast on 429, no aggressive retryCategory=Security

5. Live Smoke Tests (Opt-In)

Purpose

Detect upstream schema drift by comparing live responses against known fixtures.

Pattern

The base class is ConnectorLiveSchemaTestBase (there is no ConnectorLiveTestBase). A subclass overrides FixturesDirectory, ConnectorName, and GetTestCases() (which maps each fixture file to a live URL via the LiveSchemaTestCase record), then calls RunSchemaDriftTestsAsync() from a method marked with the [LiveTest] attribute. The [LiveTest] / [LiveTheory] attributes set Skip automatically unless STELLAOPS_LIVE_TESTS=true, so there is no Skip.IfNot(IsLiveLaneEnabled()) call. Drift is detected by FixtureUpdater.CheckDriftAsync (normalized-JSON comparison), not a hand-rolled AssertSchemaMatches.

using StellaOps.TestKit;
using StellaOps.TestKit.Connectors;
using Xunit;

[Trait("Category", TestCategories.Live)]
public sealed class NvdLiveSchemaTests : ConnectorLiveSchemaTestBase
{
    protected override string FixturesDirectory =>
        Path.Combine(AppContext.BaseDirectory, "Fixtures");

    protected override string ConnectorName => "NVD";

    protected override IEnumerable<LiveSchemaTestCase> GetTestCases()
    {
        yield return new(
            "typical-cve.json",
            "https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2024-0001",
            "Typical NVD CVE structure");
    }

    [LiveTest]
    public Task DetectSchemaDrift() => RunSchemaDriftTestsAsync();
}

Configuration

Live tests are:

6. Fixture Updater

Purpose

Refresh fixtures from live sources when upstream schemas change intentionally.

Mechanism

FixtureUpdater is a library class in StellaOps.TestKit.Connectors, not a standalone CLI. It is driven by the STELLAOPS_UPDATE_FIXTURES=true environment variable and is consumed two ways:

Public methods include UpdateFixtureFromUrlAsync, UpdateJsonFixtureFromUrlAsync, SaveFixtureAsync, SaveExpectedSnapshotAsync<T>, and CheckDriftAsync (returns a FixtureDriftReport). All write operations are no-ops unless updating is enabled.

Usage

# Refresh fixtures by running the live schema tests with both flags set.
# Drift triggers an auto-rewrite (and a warning) instead of a failure.
STELLAOPS_LIVE_TESTS=true STELLAOPS_UPDATE_FIXTURES=true \
  dotnet test --filter "Category=Live&FullyQualifiedName~Nvd"

# Regenerate parser snapshots (Expected/*.canonical.*.json) for a connector.
STELLAOPS_UPDATE_FIXTURES=true \
  dotnet test --filter "Category=Snapshot&FullyQualifiedName~Nvd"

Workflow

  1. The live schema test detects drift (CheckDriftAsync reports normalized-JSON delta).
  2. A developer re-runs with STELLAOPS_UPDATE_FIXTURES=true to rewrite the fixture.
  3. The developer reviews the diff for intentional vs accidental changes.
  4. If intentional: update the parser/mapper and commit the refreshed fixtures.
  5. If accidental: investigate the upstream API issue before committing.

7. Test Traits and CI Integration

Trait Assignment

Connector tests carry two trait families: a Lane trait consumed by the CI lane filters (see CI Lane Filters) and a Category trait drawn from StellaOps.TestKit.TestCategories. The observed assignments in src/Concelier are:

Test CategoryLane traitCategory traitPR-Gating
Parser snapshot tests[Trait("Lane", "Unit")][Trait("Category", "Snapshot")]Yes
Determinism tests[Trait("Lane", "Unit")][Trait("Category", "Determinism")]Yes
Resilience tests[Trait("Lane", "Unit")][Trait("Category", "Resilience")]Yes
Security tests—[Trait("Category", TestCategories.Unit)] + [Trait("Category", TestCategories.Security)]Yes
Live schema tests—[Trait("Category", TestCategories.Live)] + [LiveTest]No

The security and live examples in src/Concelier tag with Category rather than Lane. Prefer adding both a Lane and a Category trait on new tests so they route under either filter; the Live lane and Category=Live are equivalent opt-in selectors.

Running Tests

# Run connector unit-lane tests
dotnet test --filter "Lane=Unit" src/Concelier/__Tests/

# Run security tests (tagged via Category)
dotnet test --filter "Category=Security" src/Concelier/__Tests/

# Run live schema-drift tests (opt-in; gated by STELLAOPS_LIVE_TESTS)
STELLAOPS_LIVE_TESTS=true dotnet test --filter "Category=Live" src/Concelier/__Tests/

8. Connector Inventory

The connector set grows frequently, so a per-connector pass/fail matrix here rots into misinformation. Treat src/as the source of truth and enumerate the live set with:

# Advisory (Concelier) connectors and their test projects
ls src/Concelier/__Libraries/StellaOps.Concelier.Connector.*
ls src/Concelier/__Tests/StellaOps.Concelier.Connector.*.Tests

# VEX (Excititor) connectors test projects
ls src/Concelier/__Tests/StellaOps.Excititor.Connectors.*.Tests

Concelier Connectors (Advisory Sources)

These are grouped by family; the list below reflects the projects present under src/Concelier/__Libraries/ / __Connectors/ (verify against the directory before relying on completeness):

Shared connector logic lives in StellaOps.Concelier.Connector.Common.

Excititor Connectors (VEX Sources)

VEX connector test projects live alongside the Concelier tests under src/Concelier/__Tests/StellaOps.Excititor.Connectors.*.Tests:

Coverage status

A reliable per-connector coverage table is maintained as generated test-coverage evidence rather than hand-curated here; see TEST_COVERAGE_MATRIX. Verified examples of the patterns in this document include the NVD parser-snapshot + resilience tests (StellaOps.Concelier.Connector.Nvd.Tests), the GHSA security + live-schema tests (StellaOps.Concelier.Connector.Ghsa.Tests), and the RedHat CSAF Excititor live-schema test (StellaOps.Excititor.Connectors.RedHat.CSAF.Tests).

References

Source paths are repo-root-relative (the shared connector test bases live in StellaOps.TestKit.Connectors):

Related docs:

Last reconciled against src/ 2026-05-30. Originated: Sprint 5100.0007.0005.