checkId: check.agent.cluster.quorum plugin: stellaops.doctor.agent severity: fail tags: [agent, cluster, quorum, ha]

Agent Cluster Quorum

What It Checks

Verifies that the agent fleet has at least the configured minimum number of reachable active agents. The check only runs when Agent:Cluster:Enabled is true.

Semantics: fleet-as-cluster (Option A, sprint 032). Same caveat as check.agent.cluster.health — there is no orchestrator-side Raft API today. “Reachable” = Status == Active AND last heartbeat within 5 minutes. “Quorum” = configurable minimum count, default 1.

Configuration:

KeyDefaultMeaning
Doctor:Agent:Cluster:MinQuorum1Minimum reachable active agents required for Pass.

Severity:

Why It Matters

Without quorum, the agent cluster cannot elect a leader, which means no task dispatch, no failover, and potentially a complete halt of agent-driven operations. Losing quorum is often the step before a full cluster outage. Monitoring quorum proactively allows operators to add members or fix partitions before the cluster becomes non-functional.

Common Causes

How to Fix

Docker Compose

# Verify all agent containers are running
docker compose -f devops/compose/docker-compose.stella-ops.yml ps | grep agent

# Scale agents to restore quorum (minimum 3 for quorum of 2)
docker compose -f devops/compose/docker-compose.stella-ops.yml up -d --scale agent=3

Ensure cluster member list is correct in .env:

AGENT__CLUSTER__ENABLED=true
AGENT__CLUSTER__MINMEMBERS=2

Bare Metal / systemd

# Check how many cluster members are online
stella agent cluster members --status online

# If a member is down, restart it
ssh <agent-host> 'sudo systemctl restart stella-agent'

# Verify quorum status
stella agent cluster quorum

Kubernetes / Helm

# Check agent pod count vs desired
kubectl get statefulset stellaops-agent -n stellaops

# Scale up if below quorum
kubectl scale statefulset stellaops-agent --replicas=3 -n stellaops

# Check pod disruption budget
kubectl get pdb -n stellaops

Set a PodDisruptionBudget to prevent quorum loss during rollouts:

# values.yaml
agent:
  cluster:
    enabled: true
    replicas: 3
  podDisruptionBudget:
    minAvailable: 2

Verification

stella doctor run --check check.agent.cluster.quorum