checkId: check.agent.task.backlog plugin: stellaops.doctor.agent severity: warn tags: [agent, task, queue, capacity]

Task Queue Backlog

Flags when the agent fleet is falling behind on queued work, so operators can scale capacity before deployment and scan SLAs slip.

Reads IAgentTaskStore.GetQueueDepthAsync for the current tenant and evaluates two signals:

  1. Total queued tasks across the agent fleet.
  2. Age of the oldest queued task.

The check raises the higher of the two results — a large queue or an old task at the head of the queue is enough to warn or fail.

Default thresholds:

ResultQueued tasksOldest queued age
Warn255 minutes
Fail10030 minutes

Thresholds can be tuned with Doctor:Agent:TaskBacklog:WarnQueued, FailQueued, WarnOldestSeconds, and FailOldestSeconds.

Evidence collected: queued task count and the age of the oldest queued task.

Why It Matters

A growing task backlog means agents cannot keep up with incoming work. Tasks age in the queue, SLA timers expire, and users experience delayed deployments and scan results. Detecting backlog growth early allows operators to scale the fleet or prioritize the queue.

Common Causes

How to Fix

Docker Compose

# Add agent replicas to drain the queue faster
docker compose -f devops/compose/docker-compose.stella-ops.yml up -d --scale agent=3

Bare Metal / systemd

# Raise per-agent concurrency on an existing agent
stella agent config --agent-id <id> --set max_concurrent_tasks=8

# Or add a new agent to the fleet
stella agent bootstrap --name agent-03 --env production --platform linux

Kubernetes / Helm

# Scale up agent replicas to increase throughput
kubectl scale deployment stellaops-agent --replicas=3 -n stellaops

# Confirm the new pods are Ready
kubectl get pods -l app.kubernetes.io/component=agent -n stellaops

Verification

stella doctor run --check check.agent.task.backlog