checkId: check.notify.queue.health plugin: stellaops.doctor.notify severity: fail tags: [notify, queue, redis, nats, infrastructure]

Notification Queue Health

This Doctor check verifies that the Stella Ops notification event and delivery queues are healthy so that alerts actually flow from the platform to their destination channels. It is aimed at operators diagnosing missed or stalled notifications; because a broken queue causes a complete notification blackout, this is a severity-fail check.

What It Checks

Verifies that the notification event and delivery queues are healthy. The check:

The check only runs when a queue transport is configured in Notify:Queue:Transport.

Why It Matters

The notification queue is the backbone of the notification pipeline. If the event queue is unhealthy, new notification events are lost. If the delivery queue is unhealthy, pending notifications to email, Slack, Teams, and webhook channels will not be delivered. This is a severity-fail check because queue failure means complete notification blackout.

Common Causes

How to Fix

Docker Compose

For Redis/Valkey transport:

# Check Redis health
docker exec <redis-container> redis-cli ping

# Check connection string
docker exec <notify-container> env | grep Notify__Queue

# Restart Redis if needed
docker restart <redis-container>

For NATS transport:

# Check NATS server status
docker exec <nats-container> nats server ping

# Check NATS logs
docker logs <nats-container> --tail 50

Bare Metal / systemd

# Redis/Valkey
redis-cli ping
redis-cli info server

# NATS
nats server ping
systemctl status nats

Verify the connection string in appsettings.json:

{
  "Notify": {
    "Queue": {
      "Transport": "redis",
      "Redis": {
        "ConnectionString": "127.1.1.2:6379"
      }
    }
  }
}

Kubernetes / Helm

kubectl exec -it <redis-pod> -- redis-cli ping
kubectl logs <notify-pod> --tail 50 | grep -i queue

Verification

stella doctor run --check check.notify.queue.health