Scenario: long requests and cancellation
Some endpoints take seconds or minutes: report generation, imports, calls to slow systems. This page shows how the endpoint timeout, the gateway’s cap and cancellation work together, with a reports service built like the service of Getting started.
When the caller disconnects or the endpoint timeout expires, the gateway sends the service a cancel frame with the request’s correlation id, and the endpoint’s CancellationToken is cancelled. This holds for the TCP, TLS, UDP and RabbitMQ transports, and for Messaging when frame protection (Router:Frame:Envelope) is configured on both sides: its signed control lane is its only cancellation path (Transports). The in-memory test harness sends no cancel. On the ASP.NET Core bridge the token is HttpContext.RequestAborted.
Over Messaging the request runs until the endpoint’s timeout and is not delivered again when the timeout expires (Transports).
How the timeout is chosen
For each request the gateway takes the endpoint’s timeout from:
defaultTimeoutin the service’smicroservice.yaml, when the endpoint has an entry;- otherwise
TimeoutSecondson[StellaEndpoint](30 seconds when not set), or the bridge’sDefaultTimeout(30 seconds) for a bridged ASP.NET Core endpoint.
Every endpoint class and every bridged endpoint carries a timeout, so the gateway’s own Gateway:Routing:DefaultTimeout applies only to an endpoint that declares none. The gateway then caps the result at Gateway:Routing:GlobalTimeoutCap (120 seconds). When the timeout expires, the gateway answers the caller with 504 and stops waiting for the service. The answer’s title is Upstream timeout (problem type urn:stellaops:router:upstream-timeout), or Upstream streaming timeout (urn:stellaops:router:upstream-streaming-timeout) for an endpoint marked as streaming.
A long request
The report endpoint declares 60 seconds and passes the cancellation token to every wait:
/// <summary>
/// A long request. It passes the cancellation token to every await, so it stops when the caller
/// disconnects or the gateway's timeout for the endpoint expires.
/// </summary>
[StellaEndpoint("POST", "/reports", TimeoutSeconds = 60)]
public sealed class GenerateReportEndpoint(ReportActivity activity, ILogger<GenerateReportEndpoint> logger)
: IStellaEndpoint<GenerateReportRequest, GenerateReportResponse>
{
public async Task<GenerateReportResponse> HandleAsync(GenerateReportRequest request, CancellationToken cancellationToken)
{
try
{
var report = await ReportBuilder.BuildAsync(request, cancellationToken);
activity.RecordCompleted();
return report;
}
catch (OperationCanceledException)
{
activity.RecordCancelled();
logger.LogInformation("Report of {Rows} rows cancelled", request.Rows);
throw;
}
}
}
curl -i -X POST http://localhost:5080/reports \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"rows":500,"delayPerBatchMilliseconds":10}'
Expected result: 200 and {"reportId":"RPT-500","rows":500}.
A short timeout
The preview endpoint does the same work with a 2-second timeout:
/// <summary>
/// The same work with a short timeout: a preview must be quick, so the gateway answers 504 when it
/// takes longer than 2 seconds, and cancels the work.
/// </summary>
[StellaEndpoint("POST", "/reports/preview", TimeoutSeconds = 2)]
public sealed class PreviewReportEndpoint(ReportActivity activity)
: IStellaEndpoint<GenerateReportRequest, GenerateReportResponse>
{
public async Task<GenerateReportResponse> HandleAsync(GenerateReportRequest request, CancellationToken cancellationToken)
{
try
{
return await ReportBuilder.BuildAsync(request, cancellationToken);
}
catch (OperationCanceledException)
{
activity.RecordCancelled();
throw;
}
}
}
curl -i -X POST http://localhost:5080/reports/preview \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"rows":1000,"delayPerBatchMilliseconds":1000}'
Expected result: 504 after about 2 seconds, and the preview’s token is cancelled at the same time, so the service stops its work.
Longer than two minutes
An endpoint that needs more than 120 seconds needs two changes:
In the service, a longer timeout:
TimeoutSeconds = 600, ordefaultTimeout: 10minmicroservice.yaml.In the gateway, a higher cap:
{ "Gateway": { "Routing": { "GlobalTimeoutCap": "10m" } } }As an environment variable:
Gateway__Routing__GlobalTimeoutCap=10m.
A reverse proxy in front of the gateway needs a read timeout at least as long (Running behind a reverse proxy).
Shutdown during a long request
On shutdown the service reports Draining and waits up to DrainDeadline for the requests in flight. The reports service sets it above its longest timeout:
builder.Services.AddStellaMicroservice(options =>
{
options.ServiceName = "reports";
options.Version = "1.0.0";
options.Region = "demo";
options.InstanceId = builder.Configuration["Router:InstanceId"] ?? $"reports-{Environment.MachineName}";
options.HeartbeatInterval = TimeSpan.FromSeconds(10);
// On shutdown, wait for a report in progress: longer than the longest endpoint timeout (60 s).
options.DrainDeadline = TimeSpan.FromSeconds(70);
options.Routers =
[
new RouterEndpointConfig { Host = gatewayHost, Port = gatewayPort, TransportType = TransportType.Tcp }
];
});
Give the process at least that long to stop: TimeoutStopSec for systemd, the stop grace period of your container platform.
Designing long endpoints
- Pass the token to every await. A handler that does stops as soon as the gateway cancels the request; one that does not keeps running after the caller has gone.
- Make long work idempotent. A caller that gave up may retry while the first run still unwinds. Key the work by a request identifier and return the stored result for a repeat.
- For work longer than the cap, start a job. Accept the request with
202, return a job identifier, and offer a status endpoint, asGET/reports/activityhints at. The caller polls instead of holding a connection open. - Requests on one connection run at the same time. A service handles each request as it arrives; the TCP, TLS, UDP and RabbitMQ clients set no limit per connection (Messaging limits it with
MaxConcurrentRequests). Bound expensive work in the service, or with the gateway’s rate limits.
The complete endpoint code
The two endpoints above, with their request and response types and the report builder, in one file of the reports service (Endpoints/GenerateReportEndpoint.cs):
using Microsoft.Extensions.Logging;
using StellaOps.Microservice;
namespace Examples.Reports.Microservice.Endpoints;
/// <summary>A report request: how many rows, and how long each batch of 100 rows takes.</summary>
public sealed record GenerateReportRequest
{
public int Rows { get; init; } = 1_000;
public int DelayPerBatchMilliseconds { get; init; } = 100;
}
/// <summary>A finished report.</summary>
public sealed record GenerateReportResponse
{
public required string ReportId { get; init; }
public required int Rows { get; init; }
}
/// <summary>
/// A long request. It passes the cancellation token to every await, so it stops when the caller
/// disconnects or the gateway's timeout for the endpoint expires.
/// </summary>
[StellaEndpoint("POST", "/reports", TimeoutSeconds = 60)]
public sealed class GenerateReportEndpoint(ReportActivity activity, ILogger<GenerateReportEndpoint> logger)
: IStellaEndpoint<GenerateReportRequest, GenerateReportResponse>
{
public async Task<GenerateReportResponse> HandleAsync(GenerateReportRequest request, CancellationToken cancellationToken)
{
try
{
var report = await ReportBuilder.BuildAsync(request, cancellationToken);
activity.RecordCompleted();
return report;
}
catch (OperationCanceledException)
{
activity.RecordCancelled();
logger.LogInformation("Report of {Rows} rows cancelled", request.Rows);
throw;
}
}
}
/// <summary>
/// The same work with a short timeout: a preview must be quick, so the gateway answers 504 when it
/// takes longer than 2 seconds, and cancels the work.
/// </summary>
[StellaEndpoint("POST", "/reports/preview", TimeoutSeconds = 2)]
public sealed class PreviewReportEndpoint(ReportActivity activity)
: IStellaEndpoint<GenerateReportRequest, GenerateReportResponse>
{
public async Task<GenerateReportResponse> HandleAsync(GenerateReportRequest request, CancellationToken cancellationToken)
{
try
{
return await ReportBuilder.BuildAsync(request, cancellationToken);
}
catch (OperationCanceledException)
{
activity.RecordCancelled();
throw;
}
}
}
/// <summary>The report work: batches of 100 rows, each waiting for the configured delay.</summary>
internal static class ReportBuilder
{
public static async Task<GenerateReportResponse> BuildAsync(GenerateReportRequest request, CancellationToken cancellationToken)
{
for (var row = 0; row < request.Rows; row += 100)
{
await Task.Delay(request.DelayPerBatchMilliseconds, cancellationToken);
}
return new GenerateReportResponse { ReportId = $"RPT-{request.Rows}", Rows = request.Rows };
}
}
The activity counter they share (ReportActivity.cs):
namespace Examples.Reports.Microservice;
/// <summary>Counts report runs that finished and runs that were cancelled, for the activity endpoint.</summary>
public sealed class ReportActivity
{
private int _completed;
private int _cancelled;
public int Completed => Volatile.Read(ref _completed);
public int Cancelled => Volatile.Read(ref _cancelled);
public void RecordCompleted() => Interlocked.Increment(ref _completed);
public void RecordCancelled() => Interlocked.Increment(ref _cancelled);
}
Register them as Getting started shows: AddSingleton<ReportActivity>(), AddScoped<GenerateReportEndpoint>() and AddScoped<PreviewReportEndpoint>().
Pitfalls
504although the endpoint declares a long timeout. The gateway’s cap is lower. RaiseGateway:Routing:GlobalTimeoutCap.defaultTimeout: 30means 30 days. Write30s.- A proxy closes the connection first. Raise its read timeout above the gateway cap.
