Router · Messaging Transport over Valkey

Status

Purpose

Enable Gateway ↔ microservice Router traffic over an offline-friendly, Redis-compatible transport (Valkey) by using the existing Messaging transport layer:

This supports environments where direct TCP/TLS microservice connections are undesirable, and where an internal message bus is the preferred control plane.

Implementation Summary

Libraries

Gateway Integration

High-Level Flow

  1. Microservice connects via messaging transport:
    • publishes a slim HELLO message with instance identity to the gateway control queue
  2. Gateway processes HELLO:
    • registers the connection identity and requests endpoint metadata replay when needed
  3. Microservice answers the replay request:
    • publishes an EndpointsUpdate frame with endpoints, schemas, and OpenAPI metadata
  4. Gateway applies the metadata replay:
    • updates routing state, effective claims, and aggregated OpenAPI
  5. Gateway routes an HTTP request to a microservice:
    • publishes a REQUEST message to the service request queue
  6. Microservice handles request:
    • executes handler (or ASP.NET bridge) and publishes a RESPONSE message
  7. Gateway returns response to the client.

Cancellation uses a separate control path:

  1. A messaging client advertises messaging.cancel-control.v1 in HELLO and heartbeat frames only when a frame-envelope verifier is registered. Without a verifier it does not create a dedicated control consumer; the legacy request-queue path remains available.
  2. When a frame-envelope signer is registered, the gateway creates and signs a target-specific payload for every live capable connection for that service; the signed payload binds the target service and connection ID. It also publishes one unchanged CANCEL to the legacy per-service request queue whenever the live fleet includes a legacy client or no authenticated control target is available.
  3. Each capable connection independently consumes router:cancels:{service}:{connection}; a blocked request batch therefore cannot delay cancellation consumption.
  4. The client verifies both the gateway frame envelope and its signed target binding before changing cancellation state. A bounded, expiring exact-correlation tombstone suppresses a request when its cancellation overtakes it, and makes legacy/control duplicate delivery idempotent.

Messaging-specific recovery behavior:

Queue Topology (Conceptual)

The Messaging transport uses a small set of queues (names are configurable):

Per-connection naming is intentional. Reusing one per-service or per-instance stream during overlapping reconnects would allow a competing consumer to take and reject a cancellation meant for the new logical connection. Each control stream is approximately trimmed to CancelControlApproximateMaxLength entries. Valkey stream/group metadata can remain after a connection disappears because the generic queue abstraction does not currently expose safe queue deletion; operators should monitor router:cancels:* cardinality and only remove lanes proven to belong to disconnected registrations after the local retention window. This residual metadata cleanup is an operational follow-up, not a reason to share a cancellation stream between replicas.

Configuration

Gateway YAML Configuration

Gateway:
  Transports:
    Messaging:
      Enabled: true
      ConnectionString: "valkey:6379"
      Database: 0
      RequestQueueTemplate: "router:requests:{service}"
      CancelControlQueueTemplate: "router:cancels:{service}:{connection}"
      ResponseQueueName: "router:responses"
      ConsumerGroup: "router-gateway"
      RequestTimeout: "30s"
      LeaseDuration: "5m"
      BatchSize: 10
      MaxConcurrentRequests: 10
      CancelTombstoneTtl: "5m"
      MaxPendingCancellations: 10000
      CancelControlApproximateMaxLength: 1000
      HeartbeatInterval: "10s"

Router:
  Frame:
    Envelope:
      ActiveKeyId: "primary"
      Keys:
        primary: "<shared-secret-provisioned-outside-source-control>"
      ReplayWindow: "00:05:00"

The Router:Frame:Envelope section must be provisioned with compatible keys on the gateway and messaging microservices. A missing section intentionally leaves general legacy frame handling in bootstrap mode, but disables advertisement and consumption of messaging.cancel-control.v1; a gateway without a signer emits only the legacy CANCEL copy. A present but invalid section fails registration at startup.

Gateway DI Registration

// In Program.cs (already implemented)
if (bootstrapOptions.Transports.Messaging.Enabled)
{
    builder.Services.AddMessagingTransport<ValkeyTransportPlugin>(
        builder.Configuration, "Gateway:Transports:Messaging");
    builder.Services.AddMessagingTransportServer();
}

Microservice

Operational Semantics (Draft)

Security Notes

Implementation Status

Completed (Sprint 8100.0011.0003)

  1. ✅ Wire Messaging transport into Gateway:
    • start/stop MessagingTransportServer in GatewayHostedService
    • subscribe to OnHelloReceived, OnHeartbeatReceived, OnEndpointsUpdated, OnResponseReceived, OnConnectionClosed events
    • reuse routing state updates and claims store updates
  2. ✅ Extend Gateway transport client to support TransportType.Messaging for dispatch.
  3. ✅ Add config options (GatewayMessagingTransportOptions) and DI mappings.
  4. ✅ Switch messaging registration from periodic full HELLO replay to explicit ResyncRequest / EndpointsUpdate control frames.

Completed (Sprint 20260718_007)

  1. Authenticated per-connection cancellation control consumer independent of request batches.
  2. Capability-gated replica fan-out with legacy per-service dual-write for rolling compatibility.
  3. Bounded cancel-before-request tombstones and full-handler-lifetime concurrency enforcement.

Remaining Work

  1. Add deployment examples (compose/helm) for Valkey transport.
  2. Add integration tests using ValkeyFixture:
    • microservice HELLO registration via messaging
    • request dispatch + response return
  3. Validate streaming support (or document as out-of-scope).