checkId: check.agent.cluster.health plugin: stellaops.doctor.agent severity: fail tags: [agent, cluster, ha, resilience]

Agent Cluster Health

What It Checks

Monitors the health of the agent cluster when clustering is enabled. The check only runs when the configuration key Agent:Cluster:Enabled is set to true.

Semantics: fleet-as-cluster (Option A, sprint 032). There is no orchestrator-side Raft-style cluster API today. This check therefore interprets “cluster” as the registered active fleet for the tenant: a “member” is an agent whose status is not Inactive/Revoked, and “reachability” is heartbeat freshness within 5 minutes. Operators expecting a Raft consensus signal should instead consult their consensus library’s own health surface — this check is about whether the fleet is responsive.

Severity:

Why It Matters

In high-availability deployments, agents form a cluster to provide redundancy and automatic failover. If cluster health degrades – members become unreachable, leader election fails, or state sync stalls – task dispatch can stop entirely or produce split-brain scenarios where two agents execute the same task concurrently, leading to deployment conflicts.

Common Causes

How to Fix

Docker Compose

# Check cluster member containers
docker compose -f devops/compose/docker-compose.stella-ops.yml ps | grep agent

# View cluster-specific logs
docker compose -f devops/compose/docker-compose.stella-ops.yml logs agent --tail 200 | grep -i cluster

# Restart all agent containers to force re-election
docker compose -f devops/compose/docker-compose.stella-ops.yml restart agent

Set clustering configuration in your .env or compose override:

AGENT__CLUSTER__ENABLED=true
AGENT__CLUSTER__MEMBERS=agent-1:8500,agent-2:8500,agent-3:8500

Bare Metal / systemd

# Check cluster status
stella agent cluster status

# View cluster member health
stella agent cluster members

# Force leader re-election if leader is unhealthy
stella agent cluster elect --force

# Restart agent to rejoin cluster
sudo systemctl restart stella-agent

Kubernetes / Helm

# Check agent StatefulSet pods
kubectl get pods -l app.kubernetes.io/component=agent -n stellaops

# View cluster gossip logs
kubectl logs -l app.kubernetes.io/component=agent -n stellaops --tail=100 | grep -i cluster

# Helm values for clustering
# agent:
#   cluster:
#     enabled: true
#     replicas: 3
helm upgrade stellaops stellaops/stellaops --set agent.cluster.enabled=true --set agent.cluster.replicas=3

Verification

stella doctor run --check check.agent.cluster.health