checkId: check.operations.scheduler plugin: stellaops.doctor.operations severity: warn tags: [operations, scheduler, core]

Scheduler Health

This Doctor check confirms that the Stella Ops scheduler is running and firing time-based work on time. It is aimed at operators and on-call engineers who need to catch a stalled scheduler before periodic tasks – feed syncs, periodic scans, report generation – silently fall behind.

What It Checks

Evaluates the scheduler service status, scheduled jobs, and execution history:

• Service status: fail if the scheduler service is not running.
• Missed executions: warn if any scheduled job executions were missed (scheduled time passed without the job running).

Evidence collected: ServiceStatus, ScheduledJobs, MissedExecutions, LastExecution, NextExecution, CompletedToday.

This check always runs (no configuration prerequisites).

Why It Matters

The scheduler is responsible for triggering time-based operations across the platform: vulnerability database syncs, periodic scans, evidence expiration, report generation, feed updates, and more. If the scheduler is down, none of these periodic tasks run, causing data staleness across the system. Missed executions indicate that the scheduler was unable to trigger a job at its scheduled time, which can cause cascading data freshness issues.

Common Causes

• Scheduler service crashed or was not started
• Service configuration error preventing startup
• System was down during a scheduled execution time (maintenance, outage)
• Scheduler overloaded with too many concurrent scheduled jobs
• Clock skew between the scheduler and other services
• Resource exhaustion preventing the scheduler from processing triggers

How to Fix

Docker Compose

# Check scheduler/orchestrator service status
docker compose -f docker-compose.stella-ops.yml ps orchestrator

# View scheduler logs
docker compose -f docker-compose.stella-ops.yml logs --tail 200 orchestrator | grep -i "scheduler\|schedule"

# Restart the service
docker compose -f docker-compose.stella-ops.yml restart orchestrator

# Review missed executions
stella scheduler preview --missed

# Trigger catch-up for missed jobs
stella scheduler catchup --dry-run
stella scheduler catchup

Bare Metal / systemd

# Check scheduler service status
sudo systemctl status stellaops-scheduler

# Start the scheduler if stopped
sudo systemctl start stellaops-scheduler

# View scheduler logs
sudo journalctl -u stellaops-scheduler --since "4 hours ago"

# Review missed executions
stella scheduler preview --missed

# Trigger catch-up
stella scheduler catchup --dry-run

# Verify system clock is synchronized
timedatectl status

Kubernetes / Helm

# Check scheduler pod status
kubectl get pods -l app=stellaops-scheduler

# View logs for the scheduler pod
kubectl logs -l app=stellaops-scheduler --tail=200

# Restart the scheduler
kubectl rollout restart deployment stellaops-scheduler

# Check NTP synchronization in the node
kubectl exec -it <scheduler-pod> -- date -u

Set in Helm values.yaml:

scheduler:
  replicas: 1  # only one scheduler instance to avoid duplicate execution
  resources:
    limits:
      memory: 512Mi
      cpu: "0.5"
  catchupOnStart: true  # run missed jobs on startup

Verification

stella doctor run --check check.operations.scheduler

Related Checks

• check.operations.job-queue – scheduler feeds jobs into the queue
• check.operations.dead-letter – scheduler-triggered jobs that fail end up in dead letter
• check.scanner.vuln – vulnerability database sync is scheduler-driven