Troubleshooting
Symptoms, causes and fixes, grouped by where they appear. For build diagnostics STELLA001 to STELLA006, see Microservice SDK: the source generator.
The gateway does not start
| Message or symptom | Cause | Fix |
|---|---|---|
The router gateway host requires JWT bearer authentication | Authentication:Schemes:Bearer lacks a key source, an issuer or an audience | Set Authority (or MetadataAddress, or SigningKeys), ValidIssuers and ValidAudiences |
This router gateway host does not read Stella Ops platform settings, and they are present: ... | A setting this host does not use is in the configuration (Settings the host refuses) | Remove the named sections |
Gateway transport '<name>' is enabled ... but no server-side transport plugin for it is loaded | A custom host does not pass the transport’s server assembly | Reference StellaOps.Router.Transport.<Name>.Server and pass its assembly to AddRouterGatewayHost |
| An insecure-transport refusal | TCP or UDP without the switch | Set STELLAOPS_ROUTER_ALLOW_INSECURE_TRANSPORTS=true, or use TLS |
rate_limiting:for_environment is configured, but no IValkeyRateLimitStore is registered | A custom host has environment rate limits without a store | Call AddRouterValkeyRateLimitStore() |
Router Gateway local transport harnesses are restricted to Development, Testing, or Local environments | The in-memory harness in another environment | Set the environment; the harness is for tests |
Unsupported duration unit | A gateway duration with an unknown unit, such as 1d | Use ms, s, m or h |
A service does not connect
| Symptom | Cause | Fix |
|---|---|---|
Stella microservice options validation failed at start-up | A required option is missing, Version is not a semantic version, or Routers is empty | Set ServiceName, Version (1.0.0), Region and one Routers entry |
Host is not configured | The transport client has no Host | Set Host and Port on the transport client |
| Repeated reconnect attempts | Wrong host or port, or the gateway’s transport is not enabled | Compare the client’s Host and Port with Gateway:Transports:<Name> (defaults: TCP 9100, TLS 9443, UDP 5102) |
Max reconnection attempts (N) reached; reporting the transport as dead so the service connects again | The gateway was unreachable for longer than the client’s own attempts | Nothing to do: the service keeps connecting until the gateway answers (Transports) |
No messaging transport configured. Set '<section>:transport' to one of: ... at start-up | The Messaging transport is enabled, but no Messaging backend is registered for its section | Register the backend in code with AddMessagingTransport<ValkeyTransportPlugin>(configuration, "<section>") (Transports) |
| TLS handshake failure | Name mismatch, an untrusted certificate, or a missing client certificate | See Transports: TLS |
| RabbitMQ: no connection on the gateway side | The services’ NodeId differs from the gateway’s | Set the same NodeId on both sides |
The bridge publishes 0 endpoints | Every mapped endpoint is under ExcludedPathPrefixes or rejected by EndpointFilter | Check both options (ASP.NET Core bridge) |
A bridged application stops at start-up with Endpoint <METHOD> <path> has no authorization metadata | An endpoint has no authorization metadata, and OnMissingAuthorization is RequireExplicit (the default) | Add RequireAuthorization to the endpoint, or set OnMissingAuthorization (ASP.NET Core bridge) |
Requests fail
| Status and detail | Cause | Fix |
|---|---|---|
401 | No valid bearer token: missing, past its expiry time, or from another issuer or audience | Check the token’s iss, aud and exp against the gateway settings |
403 | The token lacks a claim the endpoint requires | Compare the token’s scope with the endpoint’s claims (GET /api/v1/router/endpoints lists the endpoints) |
404 Endpoint not found | No connected instance registers the method and path | Check the instance list; a service may have disconnected |
503 No instances available now and then | Instances flap between healthy and unhealthy: a service’s HeartbeatInterval is longer than the gateway’s stale threshold (30 seconds by default); the service logged a HeartbeatInterval warning at start-up | Use the SDK default (10 seconds), or raise the gateway’s Gateway:Health thresholds |
503 No instances available always | Every instance is unhealthy or draining, or the request names a version no instance has | Check the instance list and the X-Api-Version header |
504 Upstream timeout, or 504 Upstream streaming timeout for an endpoint marked as streaming | The endpoint took longer than its timeout, or the gateway’s cap (120 seconds) | Raise the endpoint timeout and Gateway:Routing:GlobalTimeoutCap together |
502 Response too large | The service’s response does not fit one frame of its transport: about 12 MiB of body over TCP and TLS (MaxFrameSize), one 8192-byte datagram over UDP, MaxMessageSize over RabbitMQ; maxBytes gives the limit | Return less per call (paging), or use a transport with a larger limit |
413 Payload Too Large on a large upload | The body is above Gateway:Routing:MaxRequestBodySize, or above what the transport carries in one frame: about 12 MiB over TCP and TLS, 8192 bytes over UDP, MaxMessageSize over RabbitMQ; maxBytes in the answer gives the limit that applied | Raise the setting, or send large files in parts (File uploads and downloads) |
500 and Handler type ... not registered in DI container in the service log | The endpoint class is not registered | AddScoped<TEndpoint>() or AddStellaEndpoint<TEndpoint>() |
400 Invalid JSON | The body is not valid JSON for the request type, on an endpoint without [ValidateSchema] | Fix the body; for a validated endpoint see What the schema requires |
422 | The body fails [ValidateSchema], or is not valid JSON on a validated endpoint (keyword json) | Read errors[].instanceLocation and errors[].keyword |
429 | A gateway or endpoint rate limit | Wait for Retry-After; see Rate limiting |
429 Per-connection limit exceeded | One caller connection has more than 100 MiB of request bodies in flight | Send fewer large requests at once on one connection; the limit is fixed |
503 Service Overloaded | The gateway has more than 1 GiB of request bodies in flight in total | Retry later, or spread large uploads over more gateways; the limit is fixed |
The work continues after the caller gave up
A gateway timeout or a caller disconnect cancels the endpoint’s CancellationToken. Work goes on when the handler does not pass the token to its awaits, or, on Messaging, when frame protection (Router:Frame:Envelope) is not configured on both sides (Long requests and cancellation).
Finding out what the gateway sees
Send GET requests with a token that carries router:routing:read to:
- the instance list,
/api/v1/router/instances: every connected instance with its service, version, region, transport, status and endpoint count; - the Endpoint list,
/api/v1/router/endpoints: the distinct endpoints.
/health needs no token and shows readiness and the licence detail:
curl -s http://localhost:5080/health
