checkId: check.agent.resource.utilization plugin: stellaops.doctor.agent severity: warn tags: [agent, resource, performance, capacity]

Agent Resource Utilization

What It Checks

Monitors CPU, memory, and disk utilization across the agent fleet so operators can relieve resource pressure before agents start failing tasks. The check reads Agent.LastResourceStatus, which every agent reports on each heartbeat — it does not probe agents directly.

For each agent it evaluates:

1. CPU utilization
2. Memory utilization (used / total)
3. Disk usage (used / total of the agent workspace volume)

Per-agent breaches are then aggregated into the overall Warn / Fail result.

Configurable thresholds (defaults shown):

This is a single-point check — it does not trend utilization over time. Pair it with your telemetry dashboards (the agent fleet exports CPU, memory, and disk metrics via the observability stack) for trend analysis.

Why It Matters

Agents that exhaust CPU, memory, or disk become unable to execute tasks reliably. CPU saturation causes task timeouts; memory exhaustion triggers OOM kills that look like intermittent crashes; disk exhaustion prevents artifact downloads and log writes. Proactive monitoring prevents these cascading failures before they impact deployment SLAs.

Common Causes

• Agent running too many concurrent tasks for its resource allocation
• Disk filled by accumulated scan artifacts, logs, or cached images
• Memory leak in long-running agent process
• Noisy neighbor on shared infrastructure consuming resources
• Resource limits not configured (no cgroup/container memory cap)

How to Fix

Docker Compose

# Check agent container resource usage
docker stats --no-stream $(docker compose -f devops/compose/docker-compose.stella-ops.yml ps -q agent)

# Set resource limits in compose override
# docker-compose.override.yml:
#   services:
#     agent:
#       deploy:
#         resources:
#           limits:
#             cpus: '2.0'
#             memory: 4G

# Clean up old task artifacts
docker compose -f devops/compose/docker-compose.stella-ops.yml exec agent \
  stella agent cleanup --older-than 7d

Bare Metal / systemd

# Check resource usage
stella agent health <agent-id>

# View system resources on agent host
top -bn1 | head -20
df -h /var/lib/stellaops

# Clean up old task artifacts
stella agent cleanup --older-than 7d

# Adjust concurrent task limit
stella agent config --agent-id <agent-id> --set max_concurrent_tasks=4

Kubernetes / Helm

# Check agent pod resource usage
kubectl top pods -l app.kubernetes.io/component=agent -n stellaops

# Set resource requests and limits in Helm values
# agent:
#   resources:
#     requests:
#       cpu: "500m"
#       memory: "1Gi"
#     limits:
#       cpu: "2000m"
#       memory: "4Gi"
helm upgrade stellaops stellaops/stellaops -f values.yaml

# Check if pods are being OOM-killed
kubectl get events -n stellaops --field-selector reason=OOMKilling

Verification

stella doctor run --check check.agent.resource.utilization

Related Checks

• check.agent.capacity – resource exhaustion reduces effective capacity
• check.agent.heartbeat.freshness – resource saturation can delay heartbeats
• check.agent.task.backlog – high utilization combined with backlog indicates need to scale