Local CI Testing Guide

Run the Stella Ops CI pipelines on your own machine before pushing, so a PR arrives green instead of bouncing off the gate.

This guide is for engineers working in the Stella Ops monorepo. It covers the local-ci.sh runner (smoke / pr / module / workflow / release / full modes), how to simulate Gitea Actions workflows with act, the opt-in test gates, and how to keep runs reliable in offline or rate-limited environments. New to the inner loop? Start with smoke, then prbefore opening a pull request.

Quick Start

Prerequisites

• Docker Desktop (Windows/macOS) or Docker Engine (Linux)
• .NET 10 SDK (download)
• Git
• Bash (Linux/macOS native, Windows via WSL2 or Git Bash)
• act (optional, for workflow simulation) - install

Installation

1. Copy environment template:

   cp devops/ci-local/.env.local.template devops/ci-local/.env.local
   

2. (Optional) Install act for workflow simulation:

   # macOS
   brew install act
   
   # Linux
   curl -sSL https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash
   
   # Windows (via Chocolatey)
   choco install act-cli
   

3. (Optional) Build CI Docker image:

   docker build -t stellaops-ci:local -f devops/docker/Dockerfile.ci .
   

Usage

Basic Commands

# Quick smoke test (~2 min)
./tools/scripts/dev/local-ci.sh smoke

# Smoke steps (isolate build vs unit tests)
./tools/scripts/dev/local-ci.sh smoke --smoke-step build
./tools/scripts/dev/local-ci.sh smoke --smoke-step unit
./tools/scripts/dev/local-ci.sh smoke --smoke-step unit-split
./tools/scripts/dev/local-ci.sh smoke --smoke-step unit-split --test-timeout 5m --progress-interval 60
./tools/scripts/dev/local-ci.sh smoke --smoke-step unit-split --project-start 1 --project-count 50

# Full PR-gating suite (~15 min)
./tools/scripts/dev/local-ci.sh pr

# Test specific module
./tools/scripts/dev/local-ci.sh module --module Scanner

# Auto-detect changed modules
./tools/scripts/dev/local-ci.sh module

# Simulate an active workflow via act (resolves .gitea/workflows/<name>.yml)
./tools/scripts/dev/local-ci.sh workflow --workflow dotnet-pr-tests

# Release dry-run (release mode always runs as dry-run; --dry-run is implied)
./tools/scripts/dev/local-ci.sh release --dry-run

# Full test suite (~45 min)
./tools/scripts/dev/local-ci.sh full

Windows

There is no local-ci.ps1. The full-featured runner (smoke/pr/module/ workflow/release/full) is bash-only (tools/scripts/dev/local-ci.sh). On Windows, run it via WSL 2 or Git Bash:

# Git Bash / WSL 2
./tools/scripts/dev/local-ci.sh smoke
./tools/scripts/dev/local-ci.sh pr

The only PowerShell entry point under devops/ci-local/ is run-act.ps1, which wraps act for workflow simulation only (it does not run the smoke/pr/module modes):

.\devops\ci-local\run-act.ps1 -List
.\devops\ci-local\run-act.ps1 -Workflow dotnet-pr-tests -DryRun
.\devops\ci-local\run-act.ps1 -Workflow test-architecture -Job test

run-act.ps1 -Workflow <name> resolves to .gitea\workflows\<name>.yml (active workflows only) and errors if the file is absent — so archived names such as test-matrix/build-test-deploy will not resolve through it.

Modes

smoke (~2 minutes)

Quick validation before pushing. Runs only Unit tests.

./tools/scripts/dev/local-ci.sh smoke

Optional stepwise smoke (to isolate hangs):

./tools/scripts/dev/local-ci.sh smoke --smoke-step build
./tools/scripts/dev/local-ci.sh smoke --smoke-step unit
./tools/scripts/dev/local-ci.sh smoke --smoke-step unit-split

What it does:

1. Builds the solution
2. Runs Unit tests
3. Reports pass/fail

pr (~15 minutes)

Full PR-gating suite matching CI exactly. Run this before opening a PR.

./tools/scripts/dev/local-ci.sh pr

What it does:

1. Starts CI services (postgres-test, valkey-test) unless --no-services
2. Builds the solution (dotnet build src/StellaOps.sln --configuration Release)
3. Runs the repro-bundle policy check (tools/scripts/validate/verify-repro-bundle-policy.sh)
4. Runs all PR-gating categories (dotnet test --filter Category=<cat> --no-build):
   • Unit
   • Architecture
   • Contract
   • Integration
   • Security
   • Golden
5. If src/Web/ changed (detected via git diff against main), also runs the Web PR-gating tests
6. Generates TRX reports under out/local-ci/trx/
7. Stops CI services (unless --keep-services)

The build does not pass an explicit warnings-as-errors flag here; warning-as-error behaviour, where enabled, comes from the projects’ own Directory.Build.props/csproj settings, not from local-ci.sh.

module

Test only the modules you’ve changed. Detects changes via git diff.

# Auto-detect changed modules
./tools/scripts/dev/local-ci.sh module

# Test specific module
./tools/scripts/dev/local-ci.sh module --module Scanner
./tools/scripts/dev/local-ci.sh module --module Concelier
./tools/scripts/dev/local-ci.sh module --module Authority

Available modules (keys of MODULE_PATHS in tools/scripts/lib/ci-common.sh): Scanner, Concelier, Authority, Policy, Attestor, EvidenceLocker, ExportCenter, Findings, SbomService, Notify, Router, Cryptography, AirGap, Cli, AdvisoryAI, ReachGraph, Orchestrator, PacksRegistry, Replay, Aoc, IssuerDirectory, Telemetry, Signals

Two of those are Node modules handled separately (npm, not dotnet test): Web and DevPortal (NODE_MODULES). Note that some module keys fan out to sibling paths — e.g. Scanner also tests src/BinaryIndex, Concelier also tests src/Excititor, Policy also tests src/RiskEngine, Attestor also tests src/Provenance.

workflow

Simulate a specific Gitea Actions workflow using act.

local-ci.sh workflow --workflow <name> resolves <name> to .gitea/workflows/<name>.yml (active workflows only — it does not search .gitea/workflows-archived/). Run ls .gitea/workflows/*.yml for the current set; representative active workflows include dotnet-pr-tests, test-architecture, chaos-router, and local-ci-verify.

# Simulate the .NET PR test workflow
./tools/scripts/dev/local-ci.sh workflow --workflow dotnet-pr-tests

# Simulate the architecture-rules workflow
./tools/scripts/dev/local-ci.sh workflow --workflow test-architecture

Note: test-matrix.yml and build-test-deploy.yml have been archived to .gitea/workflows-archived/. local-ci.sh workflow will not find them (it only looks in .gitea/workflows/). To dry-run an archived workflow, invoke act directly against the archive path, e.g. act -W .gitea/workflows-archived/test-matrix.yml -n (see devops/ci-local/README.md).

Requirements:

• act must be installed
• CI Docker image built (stellaops-ci:local)

release

Dry-run release pipeline to verify release artifacts.

./tools/scripts/dev/local-ci.sh release --dry-run

Release mode always runs as a dry-run for safety; if you omit --dry-run the script logs a warning and forces it on. It never publishes.

What it does:

1. Builds the solution (dotnet build src/StellaOps.sln --configuration Release)
2. Logs the CLI runtimes it would build (linux-x64, linux-arm64, osx-arm64, win-x64) — does not actually package
3. Runs helm lint devops/helm/stellaops if helm is on PATH (skipped otherwise)
4. Logs the release manifest it would generate (release notes, changelog, compose files, SBOM, checksums)
5. Does NOT publish anything

full (~45 minutes)

Complete test suite including extended categories.

./tools/scripts/dev/local-ci.sh full

Categories run:

• PR-Gating: Unit, Architecture, Contract, Integration, Security, Golden
• Extended: Performance, Benchmark, AirGap, Chaos, Determinism, Resilience, Observability

Options

Test Categories

PR-Gating (Required for merge)

Extended (On-demand)

The exact extended set run by full mode is, in order: Performance, Benchmark, AirGap, Chaos, Determinism, Resilience, Observability (EXTENDED_CATEGORIES in tools/scripts/dev/local-ci.sh). The ~time values are rough estimates, not measured budgets.

Opt-in test gates (env-var-gated suites)

Some integration suites stay skipped by default because they need infrastructure (Docker, broker container) or product-side enablement work that isn’t ready yet. They surface as Skipped (not Failed) in test runs unless you opt in. Set the env var locally before running dotnet test:

Example:

# Run RabbitMQ broker-integration suite locally
STELLAOPS_TEST_RABBITMQ=1 dotnet test \
  src/Router/__Tests/StellaOps.Router.Transport.RabbitMq.Tests/StellaOps.Router.Transport.RabbitMq.Tests.csproj

# Run Router chaos suite locally
STELLAOPS_TEST_CHAOS=1 dotnet test \
  src/__Tests/chaos/StellaOps.Chaos.Router.Tests/StellaOps.Chaos.Router.Tests.csproj

CI lanes that exercise these suites set the env var explicitly — they are not left to default.

CI Services

The local CI uses Docker Compose to run required services.

Services Started

Manual Service Management

# Start services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci up -d

# Check status
docker compose -f devops/compose/docker-compose.testing.yml --profile ci ps

# View logs
docker compose -f devops/compose/docker-compose.testing.yml logs postgres-test

# Stop services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci down

# Stop and remove volumes
docker compose -f devops/compose/docker-compose.testing.yml --profile ci down -v

Configuration

Environment Variables

Copy the template and customize:

cp devops/ci-local/.env.local.template devops/ci-local/.env.local

.env.local is git-ignored and loaded by both local-ci.sh (via load_env_file) and act (via the --env-file line in .actrc). The keys below are the ones actually defined in devops/ci-local/.env.local.template:

# Test toggles
STELLAOPS_TEST_RABBITMQ=0          # 1 = enable RabbitMQ integration tests
STELLAOPS_TEST_VALKEY=0            # 1 = enable Valkey/Redis integration tests
STELLAOPS_INTEGRATION_TESTS=false  # enable full integration suite
STELLAOPS_LIVE_TESTS=false         # enable live/network-dependent tests

# External service connections (empty = Testcontainers auto-configure)
STELLAOPS_TEST_POSTGRES_CONNECTION=
STELLAOPS_TEST_MONGO_URI=

# HSM / crypto (empty = skip HSM tests)
STELLAOPS_SOFTHSM_LIB=

# .NET runtime
DOTNET_NOLOGO=1
DOTNET_CLI_TELEMETRY_OPTOUT=1
DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
TZ=UTC

Execution engine, parallelism, and verbosity are command-line flags (--docker/--native/--act, --parallel <n>, --verbose), not env vars. local-ci.sh has no LOCAL_CI_MODE/LOCAL_CI_PARALLEL/LOCAL_CI_VERBOSE or VALKEY_CONNECTION_STRING settings — use the flags from the Options table.

Act Configuration

The repo-root .actrc file configures act for workflow simulation:

# Platform mappings - use local CI image for consistent environment
--platform ubuntu-22.04=stellaops-ci:local
--platform ubuntu-latest=stellaops-ci:local

# Container architecture (amd64 for consistency)
--container-architecture linux/amd64

# Environment variables matching CI
--env DOTNET_NOLOGO=1
--env DOTNET_CLI_TELEMETRY_OPTOUT=1
--env DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
--env TZ=UTC

# Load local secrets/environment
--env-file devops/ci-local/.env.local

# Bind mount the repository (faster than copying)
--bind

# Reuse containers between runs (faster)
--reuse

# Artifact server path for uploads
--artifact-server-path ./out/act-artifacts

# Default event file
--eventpath devops/ci-local/events/pull-request.json

Note: local-ci.sh workflow builds its own act argument list (platform, env vars, --bind, event path, --env-file) rather than reading .actrc; .actrc is consumed when you invoke bare act (or run-act.sh/run-act.ps1).

Offline & Cache

Local CI can run in offline or rate-limited environments. These steps help ensure reliable builds.

NuGet Cache Warmup

Before running tests offline or during rate-limited periods, warm the NuGet cache:

# Warm cache with throttled requests to avoid 429 errors
export NUGET_MAX_HTTP_REQUESTS=4
dotnet restore src/<Module>/StellaOps.<Module>.sln --disable-parallel

# Verify cache is populated
ls ~/.nuget/packages | wc -l

# PowerShell equivalent
$env:NUGET_MAX_HTTP_REQUESTS = "4"
dotnet restore src\<Module>\StellaOps.<Module>.sln --disable-parallel

# Verify cache
(Get-ChildItem "$env:USERPROFILE\.nuget\packages").Count

See the Solution Build Guide for the module solution list.

Rate Limiting Mitigation

If encountering NuGet 429 (Too Many Requests) errors from package sources:

1. Reduce concurrency:

   export NUGET_MAX_HTTP_REQUESTS=2
   dotnet restore --disable-parallel
   

2. Retry off-peak hours (avoid UTC 09:00-17:00 on weekdays)

3. Use local cache fallback:

   # Configure fallback to local cache only (offline mode)
   dotnet restore --source ~/.nuget/packages
   

Docker Image Caching

Pre-pull required CI images to avoid network dependency during tests:

# Pull CI services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci pull

# Build local CI image
docker build -t stellaops-ci:local -f devops/docker/Dockerfile.ci .

# Verify images are cached
docker images | grep -E "stellaops|postgres|valkey|rustfs"

Offline-Safe Test Execution

For fully offline validation:

# 1. Ensure NuGet cache is warm (see above)
# 2. Start local CI services (pre-pulled)
docker compose -f devops/compose/docker-compose.testing.yml --profile ci up -d

# 3. Run smoke offline (build + Unit tests; relies on a warm NuGet cache)
#    NOTE: local-ci.sh has no --no-restore flag — restore/build are driven by the
#    underlying `dotnet build`/`dotnet test` it invokes. Warm the cache first.
./tools/scripts/dev/local-ci.sh smoke

# 4. Or run a specific category offline directly with dotnet
#    (the smoke/pr categories already pass --no-build internally)
dotnet test src/<Module>/StellaOps.<Module>.sln \
  --filter "Category=Unit" \
  --no-restore \
  --no-build

Test Fixtures for Air-Gap

Some tests require fixture data that must be present locally:

To verify fixtures are present:

find src -type d -name "Fixtures" | head -20

Troubleshooting

Docker Issues

# Reset CI services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci down -v

# Rebuild CI image
docker build --no-cache -t stellaops-ci:local -f devops/docker/Dockerfile.ci .

# Check Docker is running
docker info

Test Failures

# Run with verbose output
./tools/scripts/dev/local-ci.sh pr --verbose

# Run specific category
./tools/scripts/dev/local-ci.sh pr --category Unit

# Check logs
cat out/local-ci/logs/Unit-*.log

# Check current test project during unit-split
cat out/local-ci/active-test.txt

Build-race during multi-project sweep (added 2026-05-03)

If a sequential sweep of dotnet test invocations across many projects (e.g. the full-sweep mode of local-ci.sh) reports any of the following CSC fingerprints, the failure is almost always a build race, not a real product defect:

CSC : error CS0006: Metadata file '.../obj/Debug/net10.0/ref/<X>.dll' could not be found
CSC : error CS8034: Unable to load Analyzer assembly '...StellaOps.Determinism.Analyzers.dll'
CSC : error CS2001: Source file '.../obj/Debug/net10.0/<X>.GeneratedMSBuildEditorConfig.editorconfig' could not be found

Root cause: each dotnet test <csproj> triggers an incremental MSBuild that visits the same shared analyzer + reference-assembly outputs (Determinism.Analyzers, Localization, Plugin, Cryptography, etc.). When two adjacent invocations overlap (dotnet keeps build server processes alive), one truncates an obj/.../ref/*.dll while the other reads it.

Workaround for an ad-hoc developer sweep — pre-build the solution once, then run each test project with --no-build:

# 1. Pre-build everything in dependency order
dotnet build src/StellaOps.sln --nologo -v minimal

# 2. Per-project test runs read the artifacts produced in step 1
dotnet test path/to/Project.Tests.csproj --no-build --nologo -v minimal

Running a single dotnet test path/to/X.Tests.csproj standalone is fine and does not need the pre-build dance — the race only manifests when many invocations chain through dotnet’s persistent build server.

Canonical CI / driver-script flags (added 2026-05-03):

The two sweep runners now expose explicit pre-build modes that implement the pattern above:

Running run-test-category.sh <category> without --prebuild / --prebuilt keeps the legacy behaviour (full incremental build per project) for callers that haven’t migrated; the script logs which mode it is in so a developer who later sees a --no-build error knows immediately whether the prebuild step ran.

Optional cold-checkout restore split: set STELLAOPS_TEST_SWEEP_PREBUILD_RESTORE=1 together with --prebuild to run an explicit dotnet restore src/StellaOps.sln step before the build (and skip the implicit restore inside dotnet build). This is purely a latency-stability flag — it has no correctness impact — and exists so CI lanes on cold checkouts can surface the ~2-minute “Determining projects to restore…” pause as a distinct phase in the lane log. Leave it off for warm dev boxes.

Secondary trip-hazard — stale bin-sprint*/obj-sprint* siblings: if dotnet build src/StellaOps.sln itself fails with error CS0579: Duplicate '<X>Attribute' referencing files under obj-sprint011/ or similar, those directories are leftovers from a prior local sprint experiment and they poison the default **/*.cs Compile glob. Remove them:

find src -maxdepth 6 -type d \( -name 'bin-sprint*' -o -name 'obj-sprint*' \) -exec rm -rf {} +

Then re-run the solution build.

The canonical CI-lane pre-build pattern (and tracking sprint) lives in docs/implplan/SPRINT_20260503_001_Tools_test_sweep_build_race.md. See .memory/standards/TESTING_PRACTICES.md § “Solution-wide test runs and memory pressure” for the adjacent (but distinct) memory-pressure failure mode.

OutOfMemoryException / hung testhost (added 2026-05-02)

The commands below use the current Scanner solution as an example. The Concelier measurement is historical; its module-root solution is retired. See the current build index.

If a solution-wide run reports OutOfMemoryException, an unexplained testhost crash, or a project that “passed yesterday” suddenly hangs, suspect memory pressure before suspecting the tests:

# 1. Confirm the test is fine in isolation
dotnet test path/to/Project.Tests.csproj

# 2. If it passes alone, the issue is environmental.
#    Stop the compose stack (it can hold 50+ GB across 60 dotnet.exe processes)
docker compose -f devops/compose/docker-compose.stella-ops.yml stop

# 3. Rerun the solution
dotnet test src/Scanner/StellaOps.Scanner.sln

If you must keep the stack up, limit MSBuild concurrency:

dotnet test src/Scanner/StellaOps.Scanner.sln -m:1

Historical verification, 2026-05-02 (Concelier + Excititor, 79 projects): every project passes clean when run individually; the OOM only manifests with parallel execution + a running compose stack. Do not “fix” tests for an OOM that does not reproduce per-project — see .memory/standards/TESTING_PRACTICES.md § “Solution-wide test runs and memory pressure”.

Act Issues

# List available workflows
act -l

# Dry run an active workflow
act -n pull_request -W .gitea/workflows/dotnet-pr-tests.yml

# Dry run an archived workflow (e.g. test-matrix, now under workflows-archived/)
act -n pull_request -W .gitea/workflows-archived/test-matrix.yml

# Debug mode
act --verbose pull_request

Windows Issues

# Check WSL is installed
wsl --status

# Install WSL if needed
wsl --install

# Check Git Bash
& "C:\Program Files\Git\bin\bash.exe" --version

Results

Test results are saved to out/local-ci/:

out/local-ci/
├── trx/                    # TRX test reports
│   ├── Unit-20250128_120000.trx
│   ├── Integration-20250128_120000.trx
│   └── ...
└── logs/                   # Execution logs
    ├── Unit-20250128_120000.log
    └── ...

Viewing Results

# List TRX files
ls -la out/local-ci/trx/

# View test log
cat out/local-ci/logs/Unit-*.log | less

Examples

Pre-Push Workflow

# 1. Quick validation
./tools/scripts/dev/local-ci.sh smoke

# 2. If smoke passes, run full PR check
./tools/scripts/dev/local-ci.sh pr

# 3. Push your changes
git push origin feature/my-branch

Module Development

# 1. Make changes to Scanner module

# 2. Run module-specific tests
./tools/scripts/dev/local-ci.sh module --module Scanner

# 3. If passes, run full PR check before PR
./tools/scripts/dev/local-ci.sh pr

AdvisoryAI durable runtime coverage

AdvisoryAiDurableRuntimeTests starts the real AdvisoryAI web host against an isolated PostgreSQL database per scenario. The consolidated startup migrator, PostgreSQL stores, knowledge startup indexer and host restart assertions remain active. The test-owned Fixtures/DurableRuntimeKnowledge corpus contains one Markdown document, one OpenAPI operation and one Doctor projection. Startup writes that corpus through the real knowledge store. StartupIndexing_PersistsOnlyTheOwnedFixtureCorpus verifies the committed rows and migration ledger, so repository-wide corpus discovery cannot silently return to this fixture. Timeline audit emission is disabled for these isolated tests.

Run the complete class, then the unfiltered adjacent project through the shared slot gate:

pwsh -NoProfile -File tools/scripts/test-targeted-xunit.ps1 `
  -Project src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj `
  -Class '*AdvisoryAiDurableRuntimeTests' -BuildProjectReferences `
  -AdditionalRunnerArgument @('-diagnostics', '-showLiveOutput', '-longRunning', '15')

pwsh -NoProfile -File tools/scripts/test-targeted-xunit.ps1 `
  -Project src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj `
  -All -BuildProjectReferences `
  -AdditionalRunnerArgument @('-diagnostics', '-showLiveOutput', '-longRunning', '15')

For a bounded investigation, capture runner output and XML under the owning task’s tmp/scratch/ directory. Apply a watchdog only to the process tree started by that run. Account for slot acquisition and compilation separately from test execution. The fixture prints session creation, host startup and host disposal phases; the startup indexer also reports corpus counts and rebuild duration. Retain every class member and each skip reason. A completed slow rebuild is evidence of corpus work; a watchdog expiry alone does not identify a PostgreSQL lock, fixture disposal wait or product failure.

Debugging Test Failures

# 1. Run with verbose output
./tools/scripts/dev/local-ci.sh pr --verbose --category Unit

# 2. Check the log
cat out/local-ci/logs/Unit-*.log

# 3. Run specific test directly
pwsh tools/scripts/test-targeted-xunit.ps1 \
  -Project src/Scanner/__Tests/StellaOps.Scanner.Tests/StellaOps.Scanner.Tests.csproj \
  -Trait 'Category=Unit' -BuildProjectReferences

Related Documentation

• CI/CD Overview
• Test Strategy
• Workflow Triggers
• Path Filters