Troubleshooting

Symptoms, causes and fixes, grouped by where they appear. For build diagnostics STELLA001 to STELLA006, see Microservice SDK: the source generator.

The gateway does not start

Message or symptomCauseFix
The router gateway host requires JWT bearer authenticationAuthentication:Schemes:Bearer lacks a key source, an issuer or an audienceSet Authority (or MetadataAddress, or SigningKeys), ValidIssuers and ValidAudiences
This router gateway host does not read Stella Ops platform settings, and they are present: ...A setting this host does not use is in the configuration (Settings the host refuses)Remove the named sections
Gateway transport '<name>' is enabled ... but no server-side transport plugin for it is loadedA custom host does not pass the transport’s server assemblyReference StellaOps.Router.Transport.<Name>.Server and pass its assembly to AddRouterGatewayHost
An insecure-transport refusalTCP or UDP without the switchSet STELLAOPS_ROUTER_ALLOW_INSECURE_TRANSPORTS=true, or use TLS
rate_limiting:for_environment is configured, but no IValkeyRateLimitStore is registeredA custom host has environment rate limits without a storeCall AddRouterValkeyRateLimitStore()
Router Gateway local transport harnesses are restricted to Development, Testing, or Local environmentsThe in-memory harness in another environmentSet the environment; the harness is for tests
Unsupported duration unitA gateway duration with an unknown unit, such as 1dUse ms, s, m or h

A service does not connect

SymptomCauseFix
Stella microservice options validation failed at start-upA required option is missing, Version is not a semantic version, or Routers is emptySet ServiceName, Version (1.0.0), Region and one Routers entry
Host is not configuredThe transport client has no HostSet Host and Port on the transport client
Repeated reconnect attemptsWrong host or port, or the gateway’s transport is not enabledCompare the client’s Host and Port with Gateway:Transports:<Name> (defaults: TCP 9100, TLS 9443, UDP 5102)
Max reconnection attempts (N) reached; reporting the transport as dead so the service connects againThe gateway was unreachable for longer than the client’s own attemptsNothing to do: the service keeps connecting until the gateway answers (Transports)
No messaging transport configured. Set '<section>:transport' to one of: ... at start-upThe Messaging transport is enabled, but no Messaging backend is registered for its sectionRegister the backend in code with AddMessagingTransport<ValkeyTransportPlugin>(configuration, "<section>") (Transports)
TLS handshake failureName mismatch, an untrusted certificate, or a missing client certificateSee Transports: TLS
RabbitMQ: no connection on the gateway sideThe services’ NodeId differs from the gateway’sSet the same NodeId on both sides
The bridge publishes 0 endpointsEvery mapped endpoint is under ExcludedPathPrefixes or rejected by EndpointFilterCheck both options (ASP.NET Core bridge)
A bridged application stops at start-up with Endpoint <METHOD> <path> has no authorization metadataAn endpoint has no authorization metadata, and OnMissingAuthorization is RequireExplicit (the default)Add RequireAuthorization to the endpoint, or set OnMissingAuthorization (ASP.NET Core bridge)

Requests fail

Status and detailCauseFix
401No valid bearer token: missing, past its expiry time, or from another issuer or audienceCheck the token’s iss, aud and exp against the gateway settings
403The token lacks a claim the endpoint requiresCompare the token’s scope with the endpoint’s claims (GET /api/v1/router/endpoints lists the endpoints)
404 Endpoint not foundNo connected instance registers the method and pathCheck the instance list; a service may have disconnected
503 No instances available now and thenInstances flap between healthy and unhealthy: a service’s HeartbeatInterval is longer than the gateway’s stale threshold (30 seconds by default); the service logged a HeartbeatInterval warning at start-upUse the SDK default (10 seconds), or raise the gateway’s Gateway:Health thresholds
503 No instances available alwaysEvery instance is unhealthy or draining, or the request names a version no instance hasCheck the instance list and the X-Api-Version header
504 Upstream timeout, or 504 Upstream streaming timeout for an endpoint marked as streamingThe endpoint took longer than its timeout, or the gateway’s cap (120 seconds)Raise the endpoint timeout and Gateway:Routing:GlobalTimeoutCap together
502 Response too largeThe service’s response does not fit one frame of its transport: about 12 MiB of body over TCP and TLS (MaxFrameSize), one 8192-byte datagram over UDP, MaxMessageSize over RabbitMQ; maxBytes gives the limitReturn less per call (paging), or use a transport with a larger limit
413 Payload Too Large on a large uploadThe body is above Gateway:Routing:MaxRequestBodySize, or above what the transport carries in one frame: about 12 MiB over TCP and TLS, 8192 bytes over UDP, MaxMessageSize over RabbitMQ; maxBytes in the answer gives the limit that appliedRaise the setting, or send large files in parts (File uploads and downloads)
500 and Handler type ... not registered in DI container in the service logThe endpoint class is not registeredAddScoped<TEndpoint>() or AddStellaEndpoint<TEndpoint>()
400 Invalid JSONThe body is not valid JSON for the request type, on an endpoint without [ValidateSchema]Fix the body; for a validated endpoint see What the schema requires
422The body fails [ValidateSchema], or is not valid JSON on a validated endpoint (keyword json)Read errors[].instanceLocation and errors[].keyword
429A gateway or endpoint rate limitWait for Retry-After; see Rate limiting
429 Per-connection limit exceededOne caller connection has more than 100 MiB of request bodies in flightSend fewer large requests at once on one connection; the limit is fixed
503 Service OverloadedThe gateway has more than 1 GiB of request bodies in flight in totalRetry later, or spread large uploads over more gateways; the limit is fixed

The work continues after the caller gave up

A gateway timeout or a caller disconnect cancels the endpoint’s CancellationToken. Work goes on when the handler does not pass the token to its awaits, or, on Messaging, when frame protection (Router:Frame:Envelope) is not configured on both sides (Long requests and cancellation).

Finding out what the gateway sees

Send GET requests with a token that carries router:routing:read to:

/health needs no token and shows readiness and the licence detail:

curl -s http://localhost:5080/health