Runbook: Scanner — Out of Memory on Large Images

Use this runbook when the Stella Ops scanner worker is killed or fails while scanning large container images — typically an OS OOM-kill (exit 137) or a .NET OutOfMemoryException during layer extraction or analysis. It is written for the Platform on-call team and focuses on the two real memory-bounding knobs (Scanner:Worker:Pipeline:MaxExtractedBytes and Scanner:Worker:MaxConcurrentJobs) plus the container memory limit.

For related symptoms, see scanner-timeout.md, scanner-worker-stuck.md, and scanner-sbom-generation-failed.md.

Metadata

FieldValue
ComponentScanner (worker — StellaOps.Scanner.Worker, .NET 10)
SeverityHigh
On-call scopePlatform team
Last updated2026-05-31
Doctor checkcheck.scanner.resources (ScannerResourceUtilizationCheck)

Reconciliation note (2026-05-31, second pass): This runbook was reconciled against src/Scanner and src/Doctor and re-verified line-by-line on 2026-05-31. The scanner worker is a .NET 10 service (StellaOps.Scanner.Worker.csproj targets net10.0), not a Go binary, so OOM surfaces as an OS OOM-kill (exit 137) or a .NET OutOfMemoryException — not a Go runtime panic. Several CLI verbs and config keys in the original draft (stella scanner config, stella scanner jobs, stella scanner logs, stella scanner scale, stella scanner workers scale, worker.memory_limit, sbom.streaming_threshold, sbom.incremental_mode, cache.layer_dedup, routing.*) do not exist in source and have been removed or marked NOT IMPLEMENTED — the only scanner subcommands are download, workers get/set, secrets, and plugins. The real memory-bounding knobs are Scanner:Worker:Pipeline:MaxExtractedBytes (default 8 GiB, verified in ScannerWorkerOptions.ScanPipelineOptions) and Scanner:Worker:MaxConcurrentJobs (default 2). The check.scanner.resources Doctor probe and its memory thresholds (Fail ≥95%, Warn ≥80%), the worker metric names, and the scanner:read/scan/write/export scopes were all confirmed against source.

Symptoms

NOT IMPLEMENTED: the original draft referenced a ScannerWorkerOOM alert and a scanner_worker_restarts_total{reason="oom"} metric. Neither exists in source. The worker meter (ScannerWorkerMetrics) emits scanner_worker_jobs_failed_total, scanner_worker_jobs_completed_total, scanner_worker_job_duration_ms, scanner_worker_stage_duration_ms, and scanner_worker_queue_latency_ms (plus cache/secret/surface counters). If you wire an alert, derive it from these counters or from container restart/OOM events at the orchestration layer.

Impact

Impact TypeDescription
User-facingLarge images cannot be scanned; smaller images may still work
Data integrityNo data loss; failed scans can be retried (the lease is released and re-leased up to Scanner:Worker:Queue:MaxAttempts, default 5)
SLA impactSpecific images blocked from release pipeline

Diagnosis

Quick checks

  1. Run the Doctor resource check. This is the canonical health probe; it reads memory/CPU/worker-pool stats and fails at ≥95% memory (warns at ≥80%):

    stella doctor --check check.scanner.resources

    Caveat: the check calls GET {Scanner:Url}/api/v1/resources/stats. That endpoint is not currently implemented in StellaOps.Scanner.WebService, so on a stock deployment the check degrades to a Warn (“Cannot retrieve resource stats: HTTP 404 / Cannot check scanner resources”). Treat a Warn here as “stats unavailable”, and fall back to container-level memory inspection (docker stats / cgroup memory on the host running the worker).

  2. Inspect the failing image’s size and layer count via the OCI inspector. Large total uncompressed size or a very high layer count is the usual trigger:

    stella image inspect <image-ref> --print-layers --output json

    The inspect output enumerates layers (size, digest, media type). There is no separate stella image layers / --sort-by command — --print-layers (default on) on inspect is the layer view.

  3. Check the worker’s memory-bounding configuration. The worker reads the Scanner:Worker configuration section. The two knobs that bound memory are:

    • Scanner:Worker:Pipeline:MaxExtractedBytes — hard cap on total uncompressed bytes extracted for one image (decompression-bomb guard). Default 8 GiB (8L * 1024 * 1024 * 1024). Enforced by LayerArchiveExtractor in BuildFilesystemStageExecutor.
    • Scanner:Worker:MaxConcurrentJobs — number of jobs processed in parallel per worker process. Default 2. Each in-flight job materializes a rootfs, so peak memory scales with this value.

    These are configured via environment/appsettings overlays (e.g. Scanner__Worker__MaxConcurrentJobs=1), not via a stella scanner config CLI verb (that verb is NOT IMPLEMENTED; the worker appsettings / container env is the source of truth).

Deep diagnosis

  1. Reproduce the scan with verbose logging and read the worker logs. The worker logs the stage in flight via ScannerWorkerInstrumentation; correlate the OOM/exit-137 with the last stage to start. Memory blow-ups are almost always in the build-filesystem stage (layer extraction) or a language/native analyzer stage processing a directory with very many files (e.g. node_modules). Read the worker container logs directly (e.g. docker logs <worker> / orchestrator log aggregation) — there is no stella scanner logs command.

  2. Confirm whether the extraction guard tripped. A clean abort with the message build-filesystem: extraction exceeded the <N>-byte limit. (an InvalidOperationException from LayerArchiveExtractor) means the image exceeds Scanner:Worker:Pipeline:MaxExtractedBytes; this is the guard working as designed (the worker fails the job rather than exhausting the host). A raw OutOfMemoryException or exit 137 with no guard message means the host/cgroup limit was hit before the 8 GiB extraction cap — lower MaxConcurrentJobs or raise the container memory limit.

  3. Check the secrets stage limits if the secrets scan is enabled. When Scanner:Worker:Secrets:Enabled=true, per-file and per-job caps apply: Secrets:MaxFileSizeBytes (default 5 MB) and Secrets:MaxFilesPerJob (default 10 000). Images with huge text trees can pressure memory here.

Resolution

Immediate mitigation

  1. Reduce per-worker concurrency so peak memory drops (each job holds its own rootfs). Set in the worker’s environment/appsettings and restart the worker container:

    # appsettings / container env overlay (NOT a CLI verb):
Scanner__Worker__MaxConcurrentJobs=1
# then restart the scanner-worker container via your orchestrator

  2. Raise the worker container memory limit at the orchestration layer (compose mem_limit / Kubernetes resources / systemd MemoryMax). Stella does not own a process-internal “memory limit” setting — memory is bounded by the container limit plus the extraction/concurrency caps above. Recommend ≥4 GiB for general use; size for the largest expected image.

  3. Re-submit the failed scan. A failed lease is retried automatically up to Scanner:Worker:Queue:MaxAttempts (default 5). To re-run manually, re-issue the scan through your normal submission path (the worker pulls jobs from the queue; there is no stella scan image --retry flag).

Root cause fix

For consistently oversized images:

  1. Raise the extraction cap only if the host can back it; otherwise keep the guard and reject the image. The cap exists to protect the host from decompression bombs:

    # appsettings / container env overlay — value in bytes (example: 12 GiB)
Scanner__Worker__Pipeline__MaxExtractedBytes=12884901888

    Pair any increase with a matching bump to the container memory limit.

  2. Run a dedicated worker pool/replica set sized for large images. The CLI tracks a worker pool name today (stella scanner workers set --count <n> --pool large-images writes a local worker-config file consumed by stella scan run); actual horizontal scaling and routing of large images to that pool is an orchestration-layer concern (replica count, memory request, queue routing), not an in-product router. The Doctor check’s remediation steps (stella scanner scale --memory +2G --cpu +2, stella scanner config set MaxConcurrentJobs 2, and stella scanner workers scale --replicas 4) are surfaced as Manual actions for this reason and are NOT IMPLEMENTED as CLI verbs (the only real scanner subcommands are download, workers get/set, secrets, and plugins) — perform the scaling through your orchestrator and set MaxConcurrentJobs via the worker env/appsettings overlay.

For images with many small files (node_modules, etc.):

  1. Lower Scanner:Worker:MaxConcurrentJobs (fewer simultaneous rootfs trees in memory) and/or constrain the secrets stage file budget (Scanner:Worker:Secrets:MaxFilesPerJob). There is no sbom.incremental_mode setting (NOT IMPLEMENTED — removed from the original draft).

Roadmap / Draft: streaming SBOM generation, incremental SBOM mode, and automatic layer-dedup caching keyed by base-image digest were described in the original draft but are not present in src/Scanner today. Track them as future work rather than operational levers; do not configure the (non-existent) sbom.streaming_threshold, sbom.incremental_mode, or cache.layer_dedup keys.

Verification

# Re-run Doctor's resource check (canonical health gate)
stella doctor --check check.scanner.resources

# Re-inspect the previously failing image to confirm size/layers
stella image inspect <image-ref> --print-layers --output json

# Re-submit the scan through your normal path and confirm it completes
# (watch scanner_worker_jobs_completed_total vs scanner_worker_jobs_failed_total
#  in your metrics backend, and tail the worker container logs for the run)

Prevention