Local CI Testing Guide

Run the Stella Ops CI pipelines on your own machine before pushing, so a PR arrives green instead of bouncing off the gate.

This guide is for engineers working in the Stella Ops monorepo. It covers the local-ci.sh runner (smoke / pr / module / workflow / release / full modes), how to simulate Gitea Actions workflows with act, the opt-in test gates, and how to keep runs reliable in offline or rate-limited environments. New to the inner loop? Start with smoke, then prbefore opening a pull request.

Quick Start

Prerequisites

• Docker Desktop (Windows/macOS) or Docker Engine (Linux)
• .NET 10 SDK (download)
• Git
• Bash (Linux/macOS native, Windows via WSL2 or Git Bash)
• act (optional, for workflow simulation) - install

Installation

1. Copy environment template:

   cp devops/ci-local/.env.local.template devops/ci-local/.env.local
   

2. (Optional) Install act for workflow simulation:

   # macOS
   brew install act
   
   # Linux
   curl -sSL https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash
   
   # Windows (via Chocolatey)
   choco install act-cli
   

3. (Optional) Build CI Docker image:

   docker build -t stellaops-ci:local -f devops/docker/Dockerfile.ci .
   

Usage

Basic Commands

# Quick smoke test (~2 min)
./devops/scripts/local-ci.sh smoke

# Smoke steps (isolate build vs unit tests)
./devops/scripts/local-ci.sh smoke --smoke-step build
./devops/scripts/local-ci.sh smoke --smoke-step unit
./devops/scripts/local-ci.sh smoke --smoke-step unit-split
./devops/scripts/local-ci.sh smoke --smoke-step unit-split --test-timeout 5m --progress-interval 60
./devops/scripts/local-ci.sh smoke --smoke-step unit-split --project-start 1 --project-count 50

# Full PR-gating suite (~15 min)
./devops/scripts/local-ci.sh pr

# Test specific module
./devops/scripts/local-ci.sh module --module Scanner

# Auto-detect changed modules
./devops/scripts/local-ci.sh module

# Simulate an active workflow via act (resolves .gitea/workflows/<name>.yml)
./devops/scripts/local-ci.sh workflow --workflow dotnet-pr-tests

# Release dry-run (release mode always runs as dry-run; --dry-run is implied)
./devops/scripts/local-ci.sh release --dry-run

# Full test suite (~45 min)
./devops/scripts/local-ci.sh full

Windows

There is no local-ci.ps1. The full-featured runner (smoke/pr/module/ workflow/release/full) is bash-only (devops/scripts/local-ci.sh). On Windows, run it via WSL 2 or Git Bash:

# Git Bash / WSL 2
./devops/scripts/local-ci.sh smoke
./devops/scripts/local-ci.sh pr

The only PowerShell entry point under devops/ci-local/ is run-act.ps1, which wraps act for workflow simulation only (it does not run the smoke/pr/module modes):

.\devops\ci-local\run-act.ps1 -List
.\devops\ci-local\run-act.ps1 -Workflow dotnet-pr-tests -DryRun
.\devops\ci-local\run-act.ps1 -Workflow test-architecture -Job test

run-act.ps1 -Workflow <name> resolves to .gitea\workflows\<name>.yml (active workflows only) and errors if the file is absent — so archived names such as test-matrix/build-test-deploy will not resolve through it.

Modes

smoke (~2 minutes)

Quick validation before pushing. Runs only Unit tests.

./devops/scripts/local-ci.sh smoke

Optional stepwise smoke (to isolate hangs):

./devops/scripts/local-ci.sh smoke --smoke-step build
./devops/scripts/local-ci.sh smoke --smoke-step unit
./devops/scripts/local-ci.sh smoke --smoke-step unit-split

What it does:

1. Builds the solution
2. Runs Unit tests
3. Reports pass/fail

pr (~15 minutes)

Full PR-gating suite matching CI exactly. Run this before opening a PR.

./devops/scripts/local-ci.sh pr

What it does:

1. Starts CI services (postgres-test, valkey-test) unless --no-services
2. Builds the solution (dotnet build src/StellaOps.sln --configuration Release)
3. Runs the repro-bundle policy check (devops/tools/verify-repro-bundle-policy.sh)
4. Runs all PR-gating categories (dotnet test --filter Category=<cat> --no-build):
   • Unit
   • Architecture
   • Contract
   • Integration
   • Security
   • Golden
5. If src/Web/ changed (detected via git diff against main), also runs the Web PR-gating tests
6. Generates TRX reports under out/local-ci/trx/
7. Stops CI services (unless --keep-services)

The build does not pass an explicit warnings-as-errors flag here; warning-as-error behaviour, where enabled, comes from the projects’ own Directory.Build.props/csproj settings, not from local-ci.sh.

module

Test only the modules you’ve changed. Detects changes via git diff.

# Auto-detect changed modules
./devops/scripts/local-ci.sh module

# Test specific module
./devops/scripts/local-ci.sh module --module Scanner
./devops/scripts/local-ci.sh module --module Concelier
./devops/scripts/local-ci.sh module --module Authority

Available modules (keys of MODULE_PATHS in devops/scripts/lib/ci-common.sh): Scanner, Concelier, Authority, Policy, Attestor, EvidenceLocker, ExportCenter, Findings, SbomService, Notify, Router, Cryptography, AirGap, Cli, AdvisoryAI, ReachGraph, Orchestrator, PacksRegistry, Replay, Aoc, IssuerDirectory, Telemetry, Signals

Two of those are Node modules handled separately (npm, not dotnet test): Web and DevPortal (NODE_MODULES). Note that some module keys fan out to sibling paths — e.g. Scanner also tests src/BinaryIndex, Concelier also tests src/Excititor, Policy also tests src/RiskEngine, Attestor also tests src/Provenance, and Notify also tests src/Notifier.

workflow

Simulate a specific Gitea Actions workflow using act.

local-ci.sh workflow --workflow <name> resolves <name> to .gitea/workflows/<name>.yml (active workflows only — it does not search .gitea/workflows-archived/). Run ls .gitea/workflows/*.yml for the current set; representative active workflows include dotnet-pr-tests, test-architecture, chaos-router, and local-ci-verify.

# Simulate the .NET PR test workflow
./devops/scripts/local-ci.sh workflow --workflow dotnet-pr-tests

# Simulate the architecture-rules workflow
./devops/scripts/local-ci.sh workflow --workflow test-architecture

Note: test-matrix.yml and build-test-deploy.yml have been archived to .gitea/workflows-archived/. local-ci.sh workflow will not find them (it only looks in .gitea/workflows/). To dry-run an archived workflow, invoke act directly against the archive path, e.g. act -W .gitea/workflows-archived/test-matrix.yml -n (see devops/ci-local/README.md).

Requirements:

• act must be installed
• CI Docker image built (stellaops-ci:local)

release

Dry-run release pipeline to verify release artifacts.

./devops/scripts/local-ci.sh release --dry-run

Release mode always runs as a dry-run for safety; if you omit --dry-run the script logs a warning and forces it on. It never publishes.

What it does:

1. Builds the solution (dotnet build src/StellaOps.sln --configuration Release)
2. Logs the CLI runtimes it would build (linux-x64, linux-arm64, osx-arm64, win-x64) — does not actually package
3. Runs helm lint devops/helm/stellaops if helm is on PATH (skipped otherwise)
4. Logs the release manifest it would generate (release notes, changelog, compose files, SBOM, checksums)
5. Does NOT publish anything

full (~45 minutes)

Complete test suite including extended categories.

./devops/scripts/local-ci.sh full

Categories run:

• PR-Gating: Unit, Architecture, Contract, Integration, Security, Golden
• Extended: Performance, Benchmark, AirGap, Chaos, Determinism, Resilience, Observability

Options

Test Categories

PR-Gating (Required for merge)

Extended (On-demand)

The exact extended set run by full mode is, in order: Performance, Benchmark, AirGap, Chaos, Determinism, Resilience, Observability (EXTENDED_CATEGORIES in devops/scripts/local-ci.sh). The ~time values are rough estimates, not measured budgets.

Opt-in test gates (env-var-gated suites)

Some integration suites stay skipped by default because they need infrastructure (Docker, broker container) or product-side enablement work that isn’t ready yet. They surface as Skipped (not Failed) in test runs unless you opt in. Set the env var locally before running dotnet test:

Example:

# Run RabbitMQ broker-integration suite locally
STELLAOPS_TEST_RABBITMQ=1 dotnet test \
  src/Router/__Tests/StellaOps.Router.Transport.RabbitMq.Tests/StellaOps.Router.Transport.RabbitMq.Tests.csproj

# Run Router chaos suite locally
STELLAOPS_TEST_CHAOS=1 dotnet test \
  src/__Tests/chaos/StellaOps.Chaos.Router.Tests/StellaOps.Chaos.Router.Tests.csproj

CI lanes that exercise these suites set the env var explicitly — they are not left to default.

CI Services

The local CI uses Docker Compose to run required services.

Services Started

Manual Service Management

# Start services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci up -d

# Check status
docker compose -f devops/compose/docker-compose.testing.yml --profile ci ps

# View logs
docker compose -f devops/compose/docker-compose.testing.yml logs postgres-test

# Stop services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci down

# Stop and remove volumes
docker compose -f devops/compose/docker-compose.testing.yml --profile ci down -v

Configuration

Environment Variables

Copy the template and customize:

cp devops/ci-local/.env.local.template devops/ci-local/.env.local

.env.local is git-ignored and loaded by both local-ci.sh (via load_env_file) and act (via the --env-file line in .actrc). The keys below are the ones actually defined in devops/ci-local/.env.local.template:

# Test toggles
STELLAOPS_TEST_RABBITMQ=0          # 1 = enable RabbitMQ integration tests
STELLAOPS_TEST_VALKEY=0            # 1 = enable Valkey/Redis integration tests
STELLAOPS_INTEGRATION_TESTS=false  # enable full integration suite
STELLAOPS_LIVE_TESTS=false         # enable live/network-dependent tests

# External service connections (empty = Testcontainers auto-configure)
STELLAOPS_TEST_POSTGRES_CONNECTION=
STELLAOPS_TEST_MONGO_URI=

# HSM / crypto (empty = skip HSM tests)
STELLAOPS_SOFTHSM_LIB=

# .NET runtime
DOTNET_NOLOGO=1
DOTNET_CLI_TELEMETRY_OPTOUT=1
DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
TZ=UTC

Execution engine, parallelism, and verbosity are command-line flags (--docker/--native/--act, --parallel <n>, --verbose), not env vars. local-ci.sh has no LOCAL_CI_MODE/LOCAL_CI_PARALLEL/LOCAL_CI_VERBOSE or VALKEY_CONNECTION_STRING settings — use the flags from the Options table.

Act Configuration

The repo-root .actrc file configures act for workflow simulation:

# Platform mappings - use local CI image for consistent environment
--platform ubuntu-22.04=stellaops-ci:local
--platform ubuntu-latest=stellaops-ci:local

# Container architecture (amd64 for consistency)
--container-architecture linux/amd64

# Environment variables matching CI
--env DOTNET_NOLOGO=1
--env DOTNET_CLI_TELEMETRY_OPTOUT=1
--env DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
--env TZ=UTC

# Load local secrets/environment
--env-file devops/ci-local/.env.local

# Bind mount the repository (faster than copying)
--bind

# Reuse containers between runs (faster)
--reuse

# Artifact server path for uploads
--artifact-server-path ./out/act-artifacts

# Default event file
--eventpath devops/ci-local/events/pull-request.json

Note: local-ci.sh workflow builds its own act argument list (platform, env vars, --bind, event path, --env-file) rather than reading .actrc; .actrc is consumed when you invoke bare act (or run-act.sh/run-act.ps1).

Offline & Cache

Local CI can run in offline or rate-limited environments. These steps help ensure reliable builds.

NuGet Cache Warmup

Before running tests offline or during rate-limited periods, warm the NuGet cache:

# Warm cache with throttled requests to avoid 429 errors
export NUGET_MAX_HTTP_REQUESTS=4
dotnet restore src/<Module>/StellaOps.<Module>.sln --disable-parallel

# Verify cache is populated
ls ~/.nuget/packages | wc -l

# PowerShell equivalent
$env:NUGET_MAX_HTTP_REQUESTS = "4"
dotnet restore src\<Module>\StellaOps.<Module>.sln --disable-parallel

# Verify cache
(Get-ChildItem "$env:USERPROFILE\.nuget\packages").Count

See the Solution Build Guide for the module solution list.

Rate Limiting Mitigation

If encountering NuGet 429 (Too Many Requests) errors from package sources:

1. Reduce concurrency:

   export NUGET_MAX_HTTP_REQUESTS=2
   dotnet restore --disable-parallel
   

2. Retry off-peak hours (avoid UTC 09:00-17:00 on weekdays)

3. Use local cache fallback:

   # Configure fallback to local cache only (offline mode)
   dotnet restore --source ~/.nuget/packages
   

Docker Image Caching

Pre-pull required CI images to avoid network dependency during tests:

# Pull CI services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci pull

# Build local CI image
docker build -t stellaops-ci:local -f devops/docker/Dockerfile.ci .

# Verify images are cached
docker images | grep -E "stellaops|postgres|valkey|rustfs"

Offline-Safe Test Execution

For fully offline validation:

# 1. Ensure NuGet cache is warm (see above)
# 2. Start local CI services (pre-pulled)
docker compose -f devops/compose/docker-compose.testing.yml --profile ci up -d

# 3. Run smoke offline (build + Unit tests; relies on a warm NuGet cache)
#    NOTE: local-ci.sh has no --no-restore flag — restore/build are driven by the
#    underlying `dotnet build`/`dotnet test` it invokes. Warm the cache first.
./devops/scripts/local-ci.sh smoke

# 4. Or run a specific category offline directly with dotnet
#    (the smoke/pr categories already pass --no-build internally)
dotnet test src/<Module>/StellaOps.<Module>.sln \
  --filter "Category=Unit" \
  --no-restore \
  --no-build

Test Fixtures for Air-Gap

Some tests require fixture data that must be present locally:

To verify fixtures are present:

find src -type d -name "Fixtures" | head -20

Troubleshooting

Docker Issues

# Reset CI services
docker compose -f devops/compose/docker-compose.testing.yml --profile ci down -v

# Rebuild CI image
docker build --no-cache -t stellaops-ci:local -f devops/docker/Dockerfile.ci .

# Check Docker is running
docker info

Test Failures

# Run with verbose output
./devops/scripts/local-ci.sh pr --verbose

# Run specific category
./devops/scripts/local-ci.sh pr --category Unit

# Check logs
cat out/local-ci/logs/Unit-*.log

# Check current test project during unit-split
cat out/local-ci/active-test.txt

Build-race during multi-project sweep (added 2026-05-03)

If a sequential sweep of dotnet test invocations across many projects (e.g. the full-sweep mode of local-ci.sh) reports any of the following CSC fingerprints, the failure is almost always a build race, not a real product defect:

CSC : error CS0006: Metadata file '.../obj/Debug/net10.0/ref/<X>.dll' could not be found
CSC : error CS8034: Unable to load Analyzer assembly '...StellaOps.Determinism.Analyzers.dll'
CSC : error CS2001: Source file '.../obj/Debug/net10.0/<X>.GeneratedMSBuildEditorConfig.editorconfig' could not be found

Root cause: each dotnet test <csproj> triggers an incremental MSBuild that visits the same shared analyzer + reference-assembly outputs (Determinism.Analyzers, Localization, Plugin, Cryptography, etc.). When two adjacent invocations overlap (dotnet keeps build server processes alive), one truncates an obj/.../ref/*.dll while the other reads it.

Workaround for an ad-hoc developer sweep — pre-build the solution once, then run each test project with --no-build:

# 1. Pre-build everything in dependency order
dotnet build src/StellaOps.sln --nologo -v minimal

# 2. Per-project test runs read the artifacts produced in step 1
dotnet test path/to/Project.Tests.csproj --no-build --nologo -v minimal

Running a single dotnet test path/to/X.Tests.csproj standalone is fine and does not need the pre-build dance — the race only manifests when many invocations chain through dotnet’s persistent build server.

Canonical CI / driver-script flags (added 2026-05-03):

The two sweep runners now expose explicit pre-build modes that implement the pattern above:

Running run-test-category.sh <category> without --prebuild / --prebuilt keeps the legacy behaviour (full incremental build per project) for callers that haven’t migrated; the script logs which mode it is in so a developer who later sees a --no-build error knows immediately whether the prebuild step ran.

Optional cold-checkout restore split: set STELLAOPS_TEST_SWEEP_PREBUILD_RESTORE=1 together with --prebuild to run an explicit dotnet restore src/StellaOps.sln step before the build (and skip the implicit restore inside dotnet build). This is purely a latency-stability flag — it has no correctness impact — and exists so CI lanes on cold checkouts can surface the ~2-minute “Determining projects to restore…” pause as a distinct phase in the lane log. Leave it off for warm dev boxes.

Secondary trip-hazard — stale bin-sprint*/obj-sprint* siblings: if dotnet build src/StellaOps.sln itself fails with error CS0579: Duplicate '<X>Attribute' referencing files under obj-sprint011/ or similar, those directories are leftovers from a prior local sprint experiment and they poison the default **/*.cs Compile glob. Remove them:

find src -maxdepth 6 -type d \( -name 'bin-sprint*' -o -name 'obj-sprint*' \) -exec rm -rf {} +

Then re-run the solution build.

The canonical CI-lane pre-build pattern (and tracking sprint) lives in docs/implplan/SPRINT_20260503_001_Tools_test_sweep_build_race.md. See docs/code-of-conduct/TESTING_PRACTICES.md § “Solution-wide test runs and memory pressure” for the adjacent (but distinct) memory-pressure failure mode.

OutOfMemoryException / hung testhost (added 2026-05-02)

If a solution-wide run reports OutOfMemoryException, an unexplained testhost crash, or a project that “passed yesterday” suddenly hangs, suspect memory pressure before suspecting the tests:

# 1. Confirm the test is fine in isolation
dotnet test path/to/Project.Tests.csproj

# 2. If it passes alone, the issue is environmental.
#    Stop the compose stack (it can hold 50+ GB across 60 dotnet.exe processes)
docker compose -f devops/compose/docker-compose.stella-ops.yml stop

# 3. Rerun the solution
dotnet test src/Concelier/StellaOps.Concelier.sln

If you must keep the stack up, throttle parallelism:

dotnet test src/Concelier/StellaOps.Concelier.sln -m:1 -- \
  xunit.parallelizeAssembly=false xunit.parallelizeTestCollections=false

Verified 2026-05-02 (Concelier + Excititor, 79 projects): every project passes clean when run individually; the OOM only manifests with parallel execution + a running compose stack. Do not “fix” tests for an OOM that does not reproduce per-project — see docs/code-of-conduct/TESTING_PRACTICES.md § “Solution-wide test runs and memory pressure”.

Act Issues

# List available workflows
act -l

# Dry run an active workflow
act -n pull_request -W .gitea/workflows/dotnet-pr-tests.yml

# Dry run an archived workflow (e.g. test-matrix, now under workflows-archived/)
act -n pull_request -W .gitea/workflows-archived/test-matrix.yml

# Debug mode
act --verbose pull_request

Windows Issues

# Check WSL is installed
wsl --status

# Install WSL if needed
wsl --install

# Check Git Bash
& "C:\Program Files\Git\bin\bash.exe" --version

Results

Test results are saved to out/local-ci/:

out/local-ci/
├── trx/                    # TRX test reports
│   ├── Unit-20250128_120000.trx
│   ├── Integration-20250128_120000.trx
│   └── ...
└── logs/                   # Execution logs
    ├── Unit-20250128_120000.log
    └── ...

Viewing Results

# List TRX files
ls -la out/local-ci/trx/

# View test log
cat out/local-ci/logs/Unit-*.log | less

Examples

Pre-Push Workflow

# 1. Quick validation
./devops/scripts/local-ci.sh smoke

# 2. If smoke passes, run full PR check
./devops/scripts/local-ci.sh pr

# 3. Push your changes
git push origin feature/my-branch

Module Development

# 1. Make changes to Scanner module

# 2. Run module-specific tests
./devops/scripts/local-ci.sh module --module Scanner

# 3. If passes, run full PR check before PR
./devops/scripts/local-ci.sh pr

Debugging Test Failures

# 1. Run with verbose output
./devops/scripts/local-ci.sh pr --verbose --category Unit

# 2. Check the log
cat out/local-ci/logs/Unit-*.log

# 3. Run specific test directly
dotnet test src/Scanner/__Tests/StellaOps.Scanner.Tests/StellaOps.Scanner.Tests.csproj \
  --filter "Category=Unit" \
  --verbosity detailed

Related Documentation

• CI/CD Overview
• Test Strategy
• Workflow Triggers
• Path Filters