Post-Incident Testing Guide

Audience: Stella Ops developers, QA engineers, and incident responders.

Purpose: Turn every production incident into a permanent regression test so the same defect can never ship twice.

Overview

Every production incident should produce a permanent regression test. An incident that reached production is, by definition, a gap your tests did not cover. This guide describes the infrastructure and workflow for generating, reviewing, and maintaining post-incident tests in the Stella Ops codebase, using the StellaOps.TestKit.Incident toolkit.

Key Principles

1. Permanent Regression: Incidents that reach production indicate a gap in testing. That gap must be permanently closed.
2. Deterministic Replay: Tests are generated from replay manifests captured during the incident.
3. Severity-Gated: P1/P2 incident tests block releases; P3/P4 tests are warning-only.
4. Traceable: Every incident test links back to the incident report and fix.

Workflow

1. Incident Triggers Replay Capture

When an incident occurs, the replay infrastructure automatically captures:

• Event sequences with correlation IDs
• Input data (sanitized for PII)
• System state at time of incident
• Configuration and policy digests

This produces a replay manifest stored in the Evidence Locker.

2. Generate Test Scaffold

Use the IncidentTestGenerator to create a test scaffold from the replay manifest:

using StellaOps.TestKit.Incident;

// Load the replay manifest
var manifestJson = File.ReadAllText("incident-replay-manifest.json");

// Create incident metadata
var metadata = new IncidentMetadata
{
    IncidentId = "INC-2026-001",
    OccurredAt = DateTimeOffset.Parse("2026-01-15T10:30:00Z"),
    RootCause = "Race condition in concurrent bundle creation",
    AffectedModules = ["EvidenceLocker", "Policy"],
    Severity = IncidentSeverity.P1,
    Title = "Evidence bundle duplication in high-concurrency scenario",
    ReportUrl = "https://incidents.stella-ops.internal/INC-2026-001"
};

// Generate the test scaffold
var generator = new IncidentTestGenerator();
var scaffold = generator.GenerateFromManifestJson(manifestJson, metadata);

// Output the generated test code
var code = scaffold.GenerateTestCode();
File.WriteAllText($"Tests/{scaffold.TestClassName}.cs", code);

3. Review and Complete Test

The generated scaffold is a starting point. A human must:

1. Review fixtures: Ensure input data is appropriate and sanitized.
2. Complete assertions: Add specific assertions for the expected behavior.
3. Verify determinism: Ensure the test produces consistent results.
4. Add to CI: Include the test in the appropriate test project.

4. Register for Tracking

Register the incident test for reporting:

generator.RegisterIncidentTest(metadata.IncidentId, scaffold);

// Generate a summary report
var report = generator.GenerateReport();
Console.WriteLine($"Total incident tests: {report.TotalTests}");
Console.WriteLine($"P1 tests: {report.BySeveority.GetValueOrDefault(IncidentSeverity.P1, 0)}");

The BySeveority property name mirrors the current IncidentTestReport API exactly (the spelling is a known typo in source). Use it as written until the API is corrected.

Incident Metadata

The IncidentMetadata record captures essential incident context:

Severity Levels

Generated Test Structure

The scaffold generates a test class with:

[Trait("Category", TestCategories.PostIncident)]
[Trait("Incident", "INC-2026-001")]
[Trait("Severity", "P1")]
public sealed class Incident_INC_2026_001_Tests
{
    private static readonly IncidentMetadata Incident = new()
    {
        IncidentId = "INC-2026-001",
        OccurredAt = DateTimeOffset.Parse("2026-01-15T10:30:00Z"),
        RootCause = "Race condition in concurrent bundle creation",
        AffectedModules = ["EvidenceLocker", "Policy"],
        Severity = IncidentSeverity.P1,
        Title = "Evidence bundle duplication"
    };

    [Fact]
    public async Task Validates_RaceCondition_Fix()
    {
        // Arrange
        // TODO: Load fixtures from replay manifest

        // Act
        // TODO: Execute the scenario that triggered the incident

        // Assert
        // TODO: Verify the fix prevents the incident condition
    }
}

CI Integration

Test Filtering

Filter post-incident tests in CI:

# Run all post-incident tests
dotnet test --filter "Category=PostIncident"

# Run only P1/P2 tests (release-gating)
dotnet test --filter "Category=PostIncident&(Severity=P1|Severity=P2)"

# Run tests for a specific incident
dotnet test --filter "Incident=INC-2026-001"

# Run tests for a specific module
dotnet test --filter "Category=PostIncident&Module:EvidenceLocker=true"

CI Lanes

Example CI Configuration

# .gitea/workflows/post-incident-tests.yml
name: Post-Incident Tests
on:
  pull_request:
  release:
    types: [created]

jobs:
  post-incident:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '10.0.x'

      - name: Run P1/P2 Incident Tests
        run: |
          dotnet test --filter "Category=PostIncident&(Severity=P1|Severity=P2)" \
            --logger "trx;LogFileName=incident-results.trx"

      - name: Upload Results
        uses: actions/upload-artifact@v4
        with:
          name: incident-test-results
          path: '**/incident-results.trx'

Best Practices

1. Sanitize Fixtures

Remove or mask any PII or sensitive data from replay fixtures:

// Before storing fixture
var sanitizedFixture = fixture
    .Replace(userEmail, "user@example.com")
    .Replace(apiKey, "REDACTED");

2. Use Deterministic Infrastructure

Ensure incident tests use TestKit’s deterministic primitives:

// Use deterministic time
using var time = new DeterministicTime(Incident.OccurredAt);

// Use deterministic random if needed
var random = new DeterministicRandom(seed: 42);

3. Document the Incident

Include comprehensive documentation in the test:

/// <summary>
/// Regression test for incident INC-2026-001: Evidence bundle duplication.
/// </summary>
/// <remarks>
/// Root cause: Race condition in concurrent bundle creation.
///
/// The incident occurred when multiple workers attempted to create the same
/// evidence bundle simultaneously. The fix added optimistic locking with
/// a unique constraint on (tenant_id, bundle_id).
///
/// Report: https://incidents.stella-ops.internal/INC-2026-001
/// Fix: PR #1234
/// </remarks>

4. Link to Sprint Tasks

Connect incident tests to the fix implementation:

[Fact]
[Trait("SprintTask", "EVIDENCE-0115-001")]
public async Task Validates_RaceCondition_Fix()

5. Evolve Tests Over Time

Incident tests may need updates as the codebase evolves:

• Update fixtures when schemas change
• Adjust assertions when behavior intentionally changes
• Add new scenarios discovered during subsequent incidents

Troubleshooting

Manifest Not Available

If the replay manifest wasn’t captured:

1. Check Evidence Locker for any captured events
2. Reconstruct the scenario from logs and metrics
3. Create a synthetic manifest for testing

Flaky Incident Tests

If the test is non-deterministic:

1. Identify non-deterministic inputs (time, random, external state)
2. Replace with TestKit deterministic primitives
3. Add retry logic only as a last resort

Test No Longer Relevant

If the fix makes the scenario impossible:

1. Document why the test is no longer applicable
2. Move to an “archived incidents” test category
3. Keep the test for documentation purposes

Related Documentation

• TestKit Usage Guide
• Testing Practices
• CI Quality Gates
• Replay Infrastructure

Changelog

v1.0 (2026-01-27)

• Initial release: IncidentTestGenerator, IncidentMetadata, TestScaffold
• CI integration patterns
• Best practices and troubleshooting