Runbook: Scanner — Worker Not Processing Jobs

Audience: Platform on-call responding to stalled or backlogged Stella Ops scan jobs.

Purpose: Diagnose and clear a Scanner worker that has stopped processing the scan queue — most commonly an out-of-memory kill, a lost Valkey/PostgreSQL connection, or a deadlocked worker pool — and restore end-to-end scan processing.

Metadata

FieldValue
ComponentScanner
SeverityCritical
On-call scopePlatform team
Last updated2026-05-31
Doctor checkscheck.scanner.queue, check.scanner.resources
Related runbooksscanner-oom · scanner-timeout · scanner-sbom-generation-failed · scanner-registry-auth

Reconciliation note (2026-05-31): This runbook was reconciled against src/ (Scanner, the Doctor scanner/storage plugins, and the CLI command factory). Several commands and signal names in the original draft did not exist in the codebase and have been corrected or flagged. See the inline NOT IMPLEMENTED callouts where a referenced capability has no backing code yet. The 2026-05-31 pass additionally: corrected the worker-concurrency config key (Scanner:Worker:MaxConcurrentJobs, not Scanner__Worker__Concurrency), added the required --from/--to arguments to stella obs logs, and flagged that the Doctor checks’ own remediation steps print non-existent CLI commands.

Symptoms

NOT IMPLEMENTED — alerts. The original draft referenced alerts ScannerWorkerStuck and ScannerQueueBacklog. Neither exists in devops/telemetry/alerts/. The only Scanner alert rules shipped today are the FN-drift SLO alerts in devops/telemetry/alerts/scanner-fn-drift-alerts.yaml (ScannerFnDriftWarning, ScannerFnDriftCritical, ScannerFnDriftEngineViolation), which are unrelated to worker/queue stalls. A worker-stall / queue-backlog alert is a prevention gap (see Prevention).

NOT IMPLEMENTED — metric. The draft metric scanner_jobs_pending does not exist. The Scanner queue meter (StellaOps.Scanner.Queue → src/Scanner/__Libraries/StellaOps.Scanner.Queue/QueueMetrics.cs) emits only: scanner_queue_enqueued_total, scanner_queue_deduplicated_total, scanner_queue_ack_total, scanner_queue_retry_total, and scanner_queue_deadletter_total (each tagged by transport). Derive a “pending” signal from enqueued_total - ack_total rather than referencing a non-existent gauge.

Impact

Impact TypeDescription
User-facingNew scans cannot complete, blocking CI/CD pipelines and release gates
Data integrityNo data loss; pending jobs will resume when the worker recovers
SLA impactScan latency SLO violated if not resolved within 15 minutes

Diagnosis

Quick checks (< 2 minutes)

  1. Check Doctor diagnostics for the scanner queue:

    stella doctor --check check.scanner.queue

    This is the ScannerQueueHealthCheck (src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Scanner/Checks/ScannerQueueHealthCheck.cs). It fails when stuckJobs > 0, when queue depth ≥ 500, or when the failure rate ≥ 15%, and warns at queue depth ≥ 100 / failure rate ≥ 5% / a growing backlog.

    NOT IMPLEMENTED — backing endpoint. This check calls the Scanner web service at GET /api/v1/queue/stats, but no such endpoint is registered in src/Scanner/StellaOps.Scanner.WebService/Endpoints/. Until that endpoint ships, the check degrades to Warn (“Cannot retrieve queue stats: HTTP 404”) rather than reporting real stuck-job counts. Treat a Warn here as “telemetry unavailable”, not “healthy”, and fall back to the manual checks below. (The check is also gated by CanRun: it only runs when Scanner:Url / Services:Scanner:Url is configured, otherwise it is skipped entirely.)

    NOT IMPLEMENTED — Doctor remediation commands. When this check does fire, its built-in remediation steps print commands that do not exist in the CLI: stella scanner queue list --status stuck, stella scanner queue retry --stuck, stella scanner workers status, and stella scanner workers scale --replicas 4 (see ScannerQueueHealthCheck.cs / ScannerResourceUtilizationCheck.cs). Ignore those suggestions and follow the Resolution section below, which uses only implemented verbs and the container runtime. (Fixing the Doctor plugin’s remediation text is a separate source-side task, out of scope for this runbook.)

  2. Check scanner resource / worker-pool health:

    stella doctor --check check.scanner.resources

    This is the ScannerResourceUtilizationCheck (src/Doctor/.../Checks/ScannerResourceUtilizationCheck.cs). It reports CPU, memory, and worker-pool occupancy (active / idle / total workers) and warns when all workers are busy with zero idle capacity.

    NOT IMPLEMENTED — backing endpoint. Same caveat: this check calls GET /api/v1/resources/stats, which is not registered in the Scanner web service, so it currently degrades to Warn.

  3. Inspect the scanner container directly (authoritative fallback while the stats endpoints are unimplemented):

    docker ps --filter name=scanner
docker stats --no-stream <scanner-container>
docker logs --tail 100 <scanner-container>

    Look for: 0% CPU on the worker, OOMKilled, repeated restarts, or connection refused / timeout errors against Valkey or PostgreSQL.

Deep diagnosis

  1. Check worker process logs via the observability log surface or the container directly:

    stella obs logs --service scanner --from 2026-05-31T00:00:00Z --to 2026-05-31T01:00:00Z

    (stella obs logs is the implemented log-query verb; there is no stella scanner logs command. --from and --to are required ISO-8601 timestamps — the command will not parse without them — and you can add --level error --query timeout to narrow the window. See BuildObsCommand in src/Cli/StellaOps.Cli/Commands/CommandFactory.cs.) Or, from the host:

    docker logs --tail 200 <scanner-container> 2>&1 | grep -Ei "timeout|refused|out of memory|oom"

  2. Check Valkey connectivity (job queue transport).

    NOT IMPLEMENTED — Doctor check. The draft referenced stella doctor --check check.storage.valkey. No Valkey/Redis Doctor check exists. The Storage plugin (src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Storage/) ships only check.storage.diskspace, check.storage.evidencelocker, and check.storage.backup. Verify Valkey manually instead:

    docker exec <valkey-container> redis-cli ping   # expect: PONG

  3. Check if the worker container was OOM-killed:

    docker inspect <scanner-container> --format '{{.State.OOMKilled}} {{.State.ExitCode}}'

    Look for: true (OOMKilled), exit code 137 (SIGKILL / OOM) or 143 (SIGTERM).

  4. Check resource utilisation via the implemented observability command:

    stella obs top --service scanner

    (stella obs top streams per-service health, SLO status, and queue details. There is no stella obs metrics subcommand.) Look for: memory > 90%, CPU sustained > 95%.

Resolution

Immediate mitigation

NOT IMPLEMENTED — worker lifecycle verbs. The original draft used stella scanner workers restart, stella scanner workers inspect, stella scanner workers scale, stella scanner scale, and stella service restart scanner. None of these exist. The implemented stella scanner surface is: download, workers get, workers set, secrets …, and plugins … (see src/Cli/StellaOps.Cli/Commands/CommandFactory.cs → BuildScannerCommand). Worker restart/scaling is performed through the container runtime today.

  1. Restart the scanner service (terminates and respawns workers):

    docker compose -f devops/compose/docker-compose.stella-ops.yml restart scanner

  2. Verify workers are processing by re-running the queue check and watching the queue counters trend (ack rate should rise relative to enqueue rate):

    stella doctor --check check.scanner.queue
stella obs top --service scanner --refresh 5

Root cause fix

If workers were OOM-killed:

  1. Raise the scanner container memory limit in the compose / deployment manifest (e.g. mem_limit / deploy.resources.limits.memory) and redeploy. There is no stella scanner config set worker.memory_limit command.

  2. Reduce concurrent scans by lowering the worker count:

    stella scanner workers set --count 2

    then restart the scanner container so the new configuration is picked up. (stella scanner workers set writes the worker count/pool config — --count is required, --pool defaults to default. There is no stella scanner config set worker.concurrency command.)

    CONFIG GAP — verify wiring. stella scanner workers set persists a local CLI config file (STELLAOPS_CLI_WORKERS_CONFIG). The CLI only feeds that value to the worker as a --workers argument on stella scan run (see CommandFactory.cs, where LoadScannerWorkerConfig().IsConfigured appends --workers <count>); a long-running scanner service started outside scan run does not read that file. To cap concurrency on the service itself, set the worker’s MaxConcurrentJobs (config section Scanner:Worker, env-var form Scanner__Worker__MaxConcurrentJobs; default 2) — see src/Scanner/StellaOps.Scanner.Worker/Options/ScannerWorkerOptions.cs. The env var Scanner__Worker__Concurrency referenced in earlier drafts does not exist; MaxConcurrentJobs is the real key.

If the Valkey connection failed:

  1. Verify Valkey health manually (no Doctor check exists — see Deep diagnosis):

    docker exec <valkey-container> redis-cli ping

  2. Restart Valkey if needed, then restart the scanner so it re-establishes the queue transport.

    NOT IMPLEMENTED — referenced runbook. The draft pointed to valkey-connection-failure.md, which does not exist in this runbook set. Cross-reference removed; create that runbook before re-linking it.

If workers appear deadlocked:

  1. Capture diagnostics, then restart the scanner container to clear the stall:

    docker logs --tail 500 <scanner-container>
docker compose -f devops/compose/docker-compose.stella-ops.yml restart scanner

    NOT IMPLEMENTED — deadlock toggle. The draft stella scanner config set worker.deadlock_detection true has no backing command or config key in src/. Removed pending implementation.

Verification

# Verify the queue and resource checks (telemetry-dependent — see endpoint caveats above)
stella doctor --check check.scanner.queue
stella doctor --check check.scanner.resources

# Submit a real scan to confirm end-to-end processing
stella scan run --runner docker --entry <scanner-image> --target <path>

# Watch queue counters / worker health drain the backlog
stella obs top --service scanner --refresh 5

# Verify no errors in recent logs (--from/--to are required ISO-8601 timestamps)
stella obs logs --service scanner --level error --from 2026-05-31T00:00:00Z --to 2026-05-31T01:00:00Z

NOT IMPLEMENTED — test-scan shortcut. The draft stella scan image --image alpine:latest --dry-run does not match the CLI. The implemented stella scan image group only exposes inspect and layers (and those currently print placeholder output). Use stella scan run … for a real scan, or stella scan image inspect --ref <ref> for metadata only.

Prevention