Router Rate Limiting Runbook

Last updated: 2026-05-30

Audience: platform operators rolling out, tuning, or troubleshooting Stella Ops Router rate limiting. For the configuration schema and field reference, see Router rate-limiting configuration.

Purpose

• Enforce centralized admission control at the Router (429 + Retry-After).
• Reduce duplicate per-service HTTP throttling and standardize response semantics.
• Keep the platform available under dependency failures (Valkey fail-open + circuit breaker).

Preconditions

• Router rate limiting configured under rate_limiting (see docs/modules/router/guides/rate-limiting-config.md).
• If for_environment is enabled:
  • Valkey reachable from Router instances.
  • Circuit breaker parameters reviewed for the environment.

Rollout plan (recommended)

1. Dry-run wiring: enable rate limiting with limits set far above peak traffic to validate middleware order, headers, and metrics.
2. Soft limits: set limits to ~2× peak traffic and monitor rejected rate and latency.
3. Production limits: set limits to target SLO and operational constraints.
4. Migration cleanup: remove any remaining service-level HTTP rate limiters to avoid double-limiting.

Response semantics (what a rejected client sees)

When a request is denied the Router responds with 429 Too Many Requests and:

• Retry-After: <seconds> — always present on a 429 (minimum 1 second, even when the triggering decision has no computed backoff).
• X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset (Unix seconds) — emitted only when the decision carries a concrete window+limit. For allowed requests these headers reflect the smallest-window rule for low-cardinality output, not a full multi-rule snapshot.
• A JSON body: { "error": "rate_limit_exceeded", "message": "...", "retryAfter": <int>, "limit": <long>, "current": <long>, "window": <int>, "scope": "instance"|"environment" }.

The scope field/tag is the rate-limit scope (instance vs environment), not an OAuth scope. The rate-limit middleware performs no scope/permission authorization of its own.

Tenant partitioning

The limiter key is partitioned by tenant when the request is authenticated: the gateway reads the first of the stellaops:tenant, tenant, or tid claims and prefixes the partition as <tenant>|<microservice>. Anonymous / tenant-less requests share a default partition keyed by microservice alone. Consequence: one tenant’s burst cannot 429 another tenant’s traffic to the same microservice. The tenant prefix is opaque to the counters — it is just part of the key.

Monitoring

Key metrics (OpenTelemetry)

• stellaops.router.ratelimit.allowed{scope,microservice,route?} — route tag added only for route-matched environment decisions.
• stellaops.router.ratelimit.rejected{scope,microservice} — incremented on the final reject; route is not tagged on the rejection path (it is recorded via the per-decision allowed/rejected counter where the route tag can apply).
• stellaops.router.ratelimit.check_latency{scope} (unit: ms) — in practice only emitted with scope=environment; instance checks are in-memory and do not record latency.
• stellaops.router.ratelimit.valkey.errors{error_type} — error_type is the exception type name.
• stellaops.router.ratelimit.circuit_breaker.trips{reason} — reason is currently always open (recorded when a check is skipped because the breaker is open).
• stellaops.router.ratelimit.instance.current (observable gauge)
• stellaops.router.ratelimit.environment.current (observable gauge)

Caveat — rejected is double-counted. A denied request increments stellaops.router.ratelimit.rejected twice: once on the per-decision path (RateLimitService.RecordDecision(..., allowed: false)) and again on the middleware reject path (RateLimitMiddleware → RecordRejection). allowed increments only once. Treat rejected as ~2× the true reject count and use it for trend/alerting rather than as an exact request count; the deny-ratio formula below therefore overstates the ratio. (Source: RateLimitService.cs:62,90 + RateLimitMiddleware.cs:81.)

PromQL examples

• Deny ratio (by microservice) — note the rejected double-count caveat above; this overstates the true ratio:
  • sum(rate(stellaops_router_ratelimit_rejected_total[5m])) by (microservice) / (sum(rate(stellaops_router_ratelimit_allowed_total[5m])) by (microservice) + sum(rate(stellaops_router_ratelimit_rejected_total[5m])) by (microservice))
• P95 check latency (environment):
  • histogram_quantile(0.95, sum(rate(stellaops_router_ratelimit_check_latency_bucket{scope="environment"}[5m])) by (le))

Incident response

Sudden spike in 429s

• Confirm whether this is expected traffic growth or misconfiguration.
• Identify the top offenders: rejected by microservice and (optionally) route.
• If misconfigured: raise limits conservatively (2×), redeploy config, then tighten gradually.

Valkey unavailable / circuit breaker opening

• Expectation: fail-open for environment limits; instance limits (if configured) still apply.
• Check:
  • stellaops.router.ratelimit.valkey.errors
  • stellaops.router.ratelimit.circuit_breaker.trips
• Actions:
  • Restore Valkey connectivity/performance.
  • Consider temporarily increasing process_back_pressure_when_more_than_per_5min to reduce Valkey load.

Troubleshooting checklist

• [ ] Confirm rate limiting middleware is enabled and runs after endpoint resolution (microservice identity available). The middleware is only inserted when a RateLimitService is registered, and the orchestrator is always registered by AddRouterRateLimiting (acting as a no-op allow pass-through when no rules are configured) — see ApplicationBuilderExtensions.UseRateLimiting.
• [ ] Validate YAML binding. Note: only invalid values fail fast at startup (e.g. negative windows, missing valkey_connection, an invalid regex pattern, throwing from Validate). Unknown/typo keys are silently ignored by IConfiguration.Bind and will leave a limit unset rather than erroring — confirm effective rules, do not rely on a startup error. Two legacy typo aliases are deliberately tolerated: back_pressure_limtis and allow_max_bust_requests.
• [ ] Confirm Valkey connectivity from Router nodes (if for_environment enabled).
• [ ] Ensure rate limiting rules exist at some level (environment defaults, microservice overrides, or route overrides); empty rules disable enforcement (RateLimitConfig.IsEnabled / GetEffectiveRules().Count == 0).
• [ ] Validate that route names are bounded before enabling route tags in dashboards/alerts (the route tag on allowed is operator-supplied route names from config, so cardinality is bounded by your config, not by request paths).

Load testing

• Run src/__Tests/load/router-rate-limiting-load-test.js (a k6 suite) against a staging Router configured with known limits. The suite is environment-config driven: configure the limits/routes on the target Router first.
• The suite ships four named scenarios (below_limit, above_limit, route_mix, activation_gate) and asserts that every 429 carries a Retry-After header (threshold router_rate_limit_429_missing_retry_after < 0.001).
• For environment (distributed) validation, run the same suite concurrently from multiple agents/machines to simulate multiple Router instances (“Scenario B” in the suite header).
• See src/__Tests/load/README.md for env vars (BASE_URL, PATH, PATHS_JSON, METHOD, TENANT_ID, AUTH_TOKEN, RESULTS_DIR) and run examples.