checkId: check.agent.task.failure.rate plugin: stellaops.doctor.agent severity: warn tags: [agent, task, failure, reliability]

Task Failure Rate

Surfaces a rising share of failing agent tasks so operators can investigate a systemic problem before it stalls promotions and deployments.

Reads IAgentTaskStore.GetExecutionStatsAsync for the current tenant and evaluates the percentage of failed tasks in the configured lookback window.

Default thresholds:

ResultFailure rate
Warn5%
Fail20%

The default lookback window is 60 minutes. Tune with Doctor:Agent:TaskFailureRate:LookbackMinutes, WarnPercent, and FailPercent.

Evidence collected: the lookback window start, total / succeeded / failed task counts, the computed failure-rate percentage, and the threshold that was applied. If no tasks ran in the window, the check returns Pass.

Why It Matters

A rising task failure rate is an early indicator of systemic problems: infrastructure issues, misconfigured environments, expired credentials, or agent software bugs. Catching a spike early allows operators to intervene before promotion and deployment work queues stall.

Common Causes

Start by finding the common error across recent failures, then fix the underlying dependency or roll back the change that introduced it.

Docker Compose

# List recent failed tasks and look for a shared error signature
stella agent tasks --status failed --last 1h

# Inspect agent logs for the failing window
docker compose -f devops/compose/docker-compose.stella-ops.yml logs agent --tail 500

Bare Metal / systemd

# List recent failed tasks
stella agent tasks --status failed --last 1h

# Inspect agent logs for the failing window
journalctl -u stella-agent --since '1 hour ago'

Kubernetes / Helm

# List recent failed tasks
stella agent tasks --status failed --last 1h

# Inspect agent pod logs for the failing window
kubectl logs -l app.kubernetes.io/component=agent -n stellaops --tail=500

Verification

stella doctor run --check check.agent.task.failure.rate