Scenario: long requests and cancellation

Some endpoints take seconds or minutes: report generation, imports, calls to slow systems. This page shows how the endpoint timeout, the gateway’s cap and cancellation work together, with a reports service built like the service of Getting started.

When the caller disconnects or the endpoint timeout expires, the gateway sends the service a cancel frame with the request’s correlation id, and the endpoint’s CancellationToken is cancelled. This holds for the TCP, TLS, UDP and RabbitMQ transports, and for Messaging when frame protection (Router:Frame:Envelope) is configured on both sides: its signed control lane is its only cancellation path (Transports). The in-memory test harness sends no cancel. On the ASP.NET Core bridge the token is HttpContext.RequestAborted.

Over Messaging the request runs until the endpoint’s timeout and is not delivered again when the timeout expires (Transports).

How the timeout is chosen

For each request the gateway takes the endpoint’s timeout from:

  1. defaultTimeout in the service’s microservice.yaml, when the endpoint has an entry;
  2. otherwise TimeoutSeconds on [StellaEndpoint] (30 seconds when not set), or the bridge’s DefaultTimeout (30 seconds) for a bridged ASP.NET Core endpoint.

Every endpoint class and every bridged endpoint carries a timeout, so the gateway’s own Gateway:Routing:DefaultTimeout applies only to an endpoint that declares none. The gateway then caps the result at Gateway:Routing:GlobalTimeoutCap (120 seconds). When the timeout expires, the gateway answers the caller with 504 and stops waiting for the service. The answer’s title is Upstream timeout (problem type urn:stellaops:router:upstream-timeout), or Upstream streaming timeout (urn:stellaops:router:upstream-streaming-timeout) for an endpoint marked as streaming.

A long request

The report endpoint declares 60 seconds and passes the cancellation token to every wait:

/// <summary>
/// A long request. It passes the cancellation token to every await, so it stops when the caller
/// disconnects or the gateway's timeout for the endpoint expires.
/// </summary>
[StellaEndpoint("POST", "/reports", TimeoutSeconds = 60)]
public sealed class GenerateReportEndpoint(ReportActivity activity, ILogger<GenerateReportEndpoint> logger)
    : IStellaEndpoint<GenerateReportRequest, GenerateReportResponse>
{
    public async Task<GenerateReportResponse> HandleAsync(GenerateReportRequest request, CancellationToken cancellationToken)
    {
        try
        {
            var report = await ReportBuilder.BuildAsync(request, cancellationToken);
            activity.RecordCompleted();
            return report;
        }
        catch (OperationCanceledException)
        {
            activity.RecordCancelled();
            logger.LogInformation("Report of {Rows} rows cancelled", request.Rows);
            throw;
        }
    }
}
curl -i -X POST http://localhost:5080/reports \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"rows":500,"delayPerBatchMilliseconds":10}'

Expected result: 200 and {"reportId":"RPT-500","rows":500}.

A short timeout

The preview endpoint does the same work with a 2-second timeout:

/// <summary>
/// The same work with a short timeout: a preview must be quick, so the gateway answers 504 when it
/// takes longer than 2 seconds, and cancels the work.
/// </summary>
[StellaEndpoint("POST", "/reports/preview", TimeoutSeconds = 2)]
public sealed class PreviewReportEndpoint(ReportActivity activity)
    : IStellaEndpoint<GenerateReportRequest, GenerateReportResponse>
{
    public async Task<GenerateReportResponse> HandleAsync(GenerateReportRequest request, CancellationToken cancellationToken)
    {
        try
        {
            return await ReportBuilder.BuildAsync(request, cancellationToken);
        }
        catch (OperationCanceledException)
        {
            activity.RecordCancelled();
            throw;
        }
    }
}
curl -i -X POST http://localhost:5080/reports/preview \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"rows":1000,"delayPerBatchMilliseconds":1000}'

Expected result: 504 after about 2 seconds, and the preview’s token is cancelled at the same time, so the service stops its work.

Longer than two minutes

An endpoint that needs more than 120 seconds needs two changes:

  1. In the service, a longer timeout: TimeoutSeconds = 600, or defaultTimeout: 10m in microservice.yaml.

  2. In the gateway, a higher cap:

    {
      "Gateway": {
        "Routing": {
          "GlobalTimeoutCap": "10m"
        }
      }
    }
    

    As an environment variable: Gateway__Routing__GlobalTimeoutCap=10m.

A reverse proxy in front of the gateway needs a read timeout at least as long (Running behind a reverse proxy).

Shutdown during a long request

On shutdown the service reports Draining and waits up to DrainDeadline for the requests in flight. The reports service sets it above its longest timeout:

builder.Services.AddStellaMicroservice(options =>
{
    options.ServiceName = "reports";
    options.Version = "1.0.0";
    options.Region = "demo";
    options.InstanceId = builder.Configuration["Router:InstanceId"] ?? $"reports-{Environment.MachineName}";
    options.HeartbeatInterval = TimeSpan.FromSeconds(10);
    // On shutdown, wait for a report in progress: longer than the longest endpoint timeout (60 s).
    options.DrainDeadline = TimeSpan.FromSeconds(70);
    options.Routers =
    [
        new RouterEndpointConfig { Host = gatewayHost, Port = gatewayPort, TransportType = TransportType.Tcp }
    ];
});

Give the process at least that long to stop: TimeoutStopSec for systemd, the stop grace period of your container platform.

Designing long endpoints

The complete endpoint code

The two endpoints above, with their request and response types and the report builder, in one file of the reports service (Endpoints/GenerateReportEndpoint.cs):

using Microsoft.Extensions.Logging;
using StellaOps.Microservice;

namespace Examples.Reports.Microservice.Endpoints;

/// <summary>A report request: how many rows, and how long each batch of 100 rows takes.</summary>
public sealed record GenerateReportRequest
{
    public int Rows { get; init; } = 1_000;

    public int DelayPerBatchMilliseconds { get; init; } = 100;
}

/// <summary>A finished report.</summary>
public sealed record GenerateReportResponse
{
    public required string ReportId { get; init; }

    public required int Rows { get; init; }
}

/// <summary>
/// A long request. It passes the cancellation token to every await, so it stops when the caller
/// disconnects or the gateway's timeout for the endpoint expires.
/// </summary>
[StellaEndpoint("POST", "/reports", TimeoutSeconds = 60)]
public sealed class GenerateReportEndpoint(ReportActivity activity, ILogger<GenerateReportEndpoint> logger)
    : IStellaEndpoint<GenerateReportRequest, GenerateReportResponse>
{
    public async Task<GenerateReportResponse> HandleAsync(GenerateReportRequest request, CancellationToken cancellationToken)
    {
        try
        {
            var report = await ReportBuilder.BuildAsync(request, cancellationToken);
            activity.RecordCompleted();
            return report;
        }
        catch (OperationCanceledException)
        {
            activity.RecordCancelled();
            logger.LogInformation("Report of {Rows} rows cancelled", request.Rows);
            throw;
        }
    }
}

/// <summary>
/// The same work with a short timeout: a preview must be quick, so the gateway answers 504 when it
/// takes longer than 2 seconds, and cancels the work.
/// </summary>
[StellaEndpoint("POST", "/reports/preview", TimeoutSeconds = 2)]
public sealed class PreviewReportEndpoint(ReportActivity activity)
    : IStellaEndpoint<GenerateReportRequest, GenerateReportResponse>
{
    public async Task<GenerateReportResponse> HandleAsync(GenerateReportRequest request, CancellationToken cancellationToken)
    {
        try
        {
            return await ReportBuilder.BuildAsync(request, cancellationToken);
        }
        catch (OperationCanceledException)
        {
            activity.RecordCancelled();
            throw;
        }
    }
}

/// <summary>The report work: batches of 100 rows, each waiting for the configured delay.</summary>
internal static class ReportBuilder
{
    public static async Task<GenerateReportResponse> BuildAsync(GenerateReportRequest request, CancellationToken cancellationToken)
    {
        for (var row = 0; row < request.Rows; row += 100)
        {
            await Task.Delay(request.DelayPerBatchMilliseconds, cancellationToken);
        }

        return new GenerateReportResponse { ReportId = $"RPT-{request.Rows}", Rows = request.Rows };
    }
}

The activity counter they share (ReportActivity.cs):

namespace Examples.Reports.Microservice;

/// <summary>Counts report runs that finished and runs that were cancelled, for the activity endpoint.</summary>
public sealed class ReportActivity
{
    private int _completed;
    private int _cancelled;

    public int Completed => Volatile.Read(ref _completed);

    public int Cancelled => Volatile.Read(ref _cancelled);

    public void RecordCompleted() => Interlocked.Increment(ref _completed);

    public void RecordCancelled() => Interlocked.Increment(ref _cancelled);
}

Register them as Getting started shows: AddSingleton<ReportActivity>(), AddScoped<GenerateReportEndpoint>() and AddScoped<PreviewReportEndpoint>().

Pitfalls