checkId: check.servicegraph.circuitbreaker plugin: stellaops.doctor.servicegraph severity: warn tags: [servicegraph, resilience, circuit-breaker, live-state]
Circuit Breaker Status
This Doctor check reports the live circuit-breaker state exposed by the local resilience registry. It keeps threshold sanity checks as secondary evidence, but a green result now requires live state: configuration alone is not treated as proof that dependencies are healthy.
What It Checks
Reads live breaker snapshots through the read-only ICircuitBreakerStateProvider seam. Evidence includes open, half-open, and closed dependency counts plus retry and backoff counters. It also records Resilience:Enabled, BreakDurationSeconds, FailureThreshold, and SamplingDurationSeconds for threshold review.
| Condition | Result |
|---|---|
| No live state provider, provider unavailable, or no breakers reported | skip |
Any dependency breaker is open | fail |
Any dependency breaker is half-open | warn |
| Live breakers are closed but thresholds are too aggressive | warn |
| Live breakers are closed and thresholds are sane | pass |
Why It Matters
Circuit breakers protect external dependencies from retry storms. Bad thresholds either trip too aggressively or never trip when a downstream service is failing. Live breaker state shows whether the dependency is degraded right now.
Common Causes
- Resilience policies were never enabled on outgoing HTTP clients
- The service did not register the read-only resilience-state provider
- Thresholds were copied from a benchmark profile into production
- Multiple services use different resilience defaults, making failures unpredictable
How to Fix
Docker Compose
services:
doctor-web:
environment:
Resilience__Enabled: "true"
Resilience__CircuitBreaker__BreakDurationSeconds: "30"
Resilience__CircuitBreaker__FailureThreshold: "5"
Resilience__CircuitBreaker__SamplingDurationSeconds: "60"
If the check skips with ICircuitBreakerStateProvider not registered, wire the local resilience-state exporter into the Doctor host. Do not reset breakers from the check.
Bare Metal / systemd
Keep breaker settings in the same configuration source used for HTTP client registration so the service and Doctor observe the same values. Register the read-only resilience-state provider with the Doctor host process.
Kubernetes / Helm
Standardize resilience values across backend-facing workloads instead of per-pod overrides. Expose only read-only breaker state to Doctor; do not grant Doctor a breaker-reset capability.
Verification
stella doctor run --check check.servicegraph.circuitbreaker
Related Checks
check.servicegraph.backend- the breaker policy protects this path when the backend degradescheck.servicegraph.timeouts- timeout settings and breaker settings should be tuned togethercheck.servicegraph.connection-pools- pool saturation can trip or amplify breaker failures