Router Chaos Testing Runbook

Sprint: SPRINT_5100_0005_0001 Last Updated: 2025-12-22 Audience: Platform / SRE engineers operating or hardening the Stella Ops Router.

Overview

This runbook describes the chaos-testing approach for the Stella Ops Router — the admission and request-routing layer in front of the platform’s microservices. The goal is to prove three behaviours under stress:

Backpressure — the Router sheds load with 429/503 + Retry-After instead of failing silently or crashing.
Graceful degradation — dependency failures (notably Valkey) fall back cleanly rather than taking the Router down.
Recovery — the Router returns to baseline latency and throughput once load drops.

For the production rate-limiting contract (response semantics, headers, metrics), see the companion Router Rate Limiting Runbook.

Test Categories

1. Load Testing (k6)

Location: src/__Tests/load/router/

Spike Test Scenarios

Scenario	Rate	Duration	Purpose
Baseline	100 req/s	1 min	Establish normal operation
10x Spike	1000 req/s	30s	Moderate overload
50x Spike	5000 req/s	30s	Severe overload
Recovery	100 req/s	2 min	Measure recovery time

Running Load Tests

# Install k6
brew install k6  # macOS
# or
choco install k6  # Windows

# Run spike test against a local Router
k6 run src/__Tests/load/router/spike-test.js \
  -e ROUTER_URL=http://localhost:8080

# Run against a staging Router
k6 run src/__Tests/load/router/spike-test.js \
  -e ROUTER_URL=https://router.staging.example.com

# Output results to JSON
k6 run src/__Tests/load/router/spike-test.js \
  --out json=results.json

2. Backpressure Verification

Location: src/__Tests/chaos/StellaOps.Chaos.Router.Tests/BackpressureVerificationTests.cs

Tests verify:

HTTP 429 responses include Retry-After header
HTTP 503 responses include Retry-After header
Retry-After values are reasonable (1-60 seconds)
No data loss during throttling

Expected Behavior

Load Level	Expected Response	Retry-After
Normal	200 OK	N/A
High (>80% capacity)	429 Too Many Requests	1-10s
Critical (>95% capacity)	503 Service Unavailable	10-60s

3. Recovery Testing

Location: src/__Tests/chaos/StellaOps.Chaos.Router.Tests/RecoveryTests.cs

Tests verify:

Router recovers within 30 seconds after load drops
No request queue corruption
Metrics return to baseline

Recovery Thresholds

Metric	Target	Critical
P95 Recovery Time	<15s	<30s
P99 Recovery Time	<25s	<45s
Data Loss	0%	0%

4. Valkey Failure Injection

Location: src/__Tests/chaos/StellaOps.Chaos.Router.Tests/ValkeyFailureTests.cs

Tests verify router behavior when Valkey (cache/session store) fails:

Graceful degradation to stateless mode
No crashes or hangs
Proper error logging
Recovery when Valkey returns

Failure Scenarios

Scenario	Expected Behavior
Valkey unreachable	Fallback to direct processing
Valkey slow (>500ms)	Timeout and continue
Valkey returns	Resume normal caching

CI Integration

Workflow: .gitea/workflows/chaos-router.yml

The StellaOps.Chaos.Router.Tests suite runs against an in-process Gateway hosted by ChaosGatewayFactory. It is triggered:

Nightly at 02:30 UTC (sprint acceptance requires three consecutive green nightly runs before the suite is declared stable).
On pull requests that touch src/__Tests/chaos/StellaOps.Chaos.Router.Tests/** or the workflow file itself.
On demand via workflow_dispatch.

Gating flags. The suite is gated on the environment variable STELLAOPS_TEST_CHAOS=1; without it the tests skip silently (see Fixtures/ChaosFactAttribute.cs). The Valkey-flap subset is gated additionally on STELLAOPS_TEST_CHAOS_VALKEY=1 and is left off in the default lane because the in-process Gateway is not yet wired to a Valkey messaging transport (the container runs as a sidecar only).

Workflow stages

Build — compile the Router and chaos test projects.
Backpressure — run BackpressureVerificationTests.
Recovery & resilience — run RecoveryTests and (when enabled) ValkeyFailureTests.
Report — upload results as workflow artifacts.

Interpreting Results

Success Criteria

Metric	Pass	Fail
Request success rate during normal load	>=99%	<95%
Throttle response rate during spike	>0% (expected)	0% (no backpressure)
Recovery time P95	<30s	>=45s
Data loss	0%	>0%

Common Failure Patterns

No Throttling Under Load

Symptom: 0% throttled requests during 50x spike Cause: Backpressure not configured or circuit breaker disabled Fix: Check router configuration backpressure.enabled=true

Slow Recovery

Symptom: Recovery time >45s Cause: Request queue not draining properly Fix: Check maxQueueSize and drainTimeoutSeconds settings

Missing Retry-After Header

Symptom: 429/503 without Retry-After Cause: Header middleware not applied Fix: Ensure UseRetryAfterMiddleware() is in pipeline

Metrics & Dashboards

Key Metrics to Monitor

# Throttle rate
rate(http_requests_total{status="429"}[5m]) / rate(http_requests_total[5m])

# Recovery time
histogram_quantile(0.95, rate(request_recovery_seconds_bucket[5m]))

# Queue depth
router_request_queue_depth

Alert Thresholds

Alert	Condition	Severity
High Throttle Rate	throttle_rate > 10% for 5m	Warning
Extended Throttle	throttle_rate > 50% for 2m	Critical
Slow Recovery	p95_recovery > 30s	Warning
No Recovery	p99_recovery > 60s	Critical

Troubleshooting

Test Environment Setup

# Start the Router locally
docker compose up router valkey

# Verify Router health
curl http://localhost:8080/health

# Verify Valkey connection
docker exec -it valkey valkey-cli ping

Debug Mode

# Run the chaos suite with verbose logging (set the gating flag first)
STELLAOPS_TEST_CHAOS=1 dotnet test \
  src/__Tests/chaos/StellaOps.Chaos.Router.Tests/StellaOps.Chaos.Router.Tests.csproj \
  --logger "console;verbosity=detailed"

# k6 with debug output
k6 run src/__Tests/load/router/spike-test.js --verbose

References

Router Architecture
Router Rate Limiting Runbook
Router Backpressure (feature reference)