Runbook: Scanner — Scan Timeout on Complex Images

Use this runbook when scans of large or complex images fail to complete in time — jobs re-leased and eventually dead-lettered, queue visibility timeouts elapsing mid-scan, or a per-mechanism timeout (registry/HTTP forward, dotnet restore pre-step, verdict push) firing. It is written for the Platform on-call team.

There is no global scan-timeout setting in Stella Ops: timeouts are per-mechanism (queue lease, restore pre-step, matcher/verdict-push forwards), so this runbook walks through each real budget rather than a single knob. For related symptoms, see scanner-oom.mdand scanner-worker-stuck.md.

Reconciliation note (re-verified 2026-05-31; first reconciled 2026-05-30): This runbook was reconciled against src/Scanner and the Doctor/CLI source. Several commands, config keys, the Doctor check, the alert, and the metric originally cited here did not exist in the codebase. Verified facts are grounded in source; constructs that are not implemented are marked NOT IMPLEMENTED with the closest real equivalent. The 2026-05-31 pass re-checked every cited config default, metric name, Doctor check ID, scope, and CLI verb against source (all confirmed) and corrected the stella image inspect/layers claims, which the prior pass had backwards (the top-level stella image inspect is fully implemented; only stella scan image … is stubbed). See Decisions & Risks at the bottom. Ground truth files:

  • Worker config: src/Scanner/StellaOps.Scanner.Worker/Options/ScannerWorkerOptions.cs
  • WebService config: src/Scanner/StellaOps.Scanner.WebService/Options/ScannerWebServiceOptions.cs
  • Worker metrics + Meter name: src/Scanner/StellaOps.Scanner.Worker/Diagnostics/ScannerWorkerMetrics.cs, ScannerWorkerInstrumentation.cs
  • Scanner Doctor checks + plugin: src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Scanner/Checks/, ScannerDoctorPlugin.cs
  • Doctor category enum: src/__Libraries/StellaOps.Doctor/Plugins/DoctorCategory.cs (Scanner is a valid --category value)
  • CLI verbs: src/Cli/StellaOps.Cli/Commands/CommandFactory.cs, DoctorCommandGroup.cs, ImageCommandGroup.cs, CommandHandlers.Image.cs
  • Scan status / manifest REST: src/Scanner/StellaOps.Scanner.WebService/Endpoints/ScanEndpoints.cs, ManifestEndpoints.cs
  • Scopes: src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs

Metadata

FieldValue
ComponentScanner
SeverityMedium
On-call scopePlatform team
Last updated2026-05-31
Doctor checkcheck.scanner.queue, check.scanner.resources (no dedicated timeout-rate check exists)

Symptoms

NOT IMPLEMENTED — verify before citing in an alert: There is no scanner_scan_timeout_total metric and no ScannerTimeoutExceeded alert in the codebase. The scanner does not emit a single “scan operation exceeded timeout of X seconds” message from a global scan timeout — there is no global scan-timeout setting. Timeouts are per-mechanism (queue lease, restore pre-step, registry/HTTP forwards). Use the real metrics in the Verification section instead.

Impact

Impact TypeDescription
User-facingSpecific images cannot be scanned; pipeline blocked
Data integrityNo data loss; jobs are idempotent and can be retried (see queue MaxDeliveryAttempts)
SLA impactRelease pipeline delayed for affected images

Diagnosis

Quick checks

  1. Run the scanner Doctor checks:

    # Closest real checks to "is the scanner falling behind / failing":
stella doctor --check check.scanner.queue
stella doctor --check check.scanner.resources
# List all available scanner checks:
stella doctor list --category Scanner

    Valid scanner Doctor check IDs (from src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Scanner/Checks/): check.scanner.queue, check.scanner.sbom, check.scanner.vuln, check.scanner.witness.graph, check.scanner.slice.cache, check.scanner.reachability, check.scanner.resources. There is no check.scanner.timeout-rate. --category is parsed to the DoctorCategory enum, so the literal value is Scanner (the value Scanner & Reachability you see as the list heading is the plugin’s display name, not the filter token).

  2. Identify failing / stuck jobs. Job state lives in the scanner schema (Postgres). There is NO stella scanner jobs list CLI command. Query the backend instead:

    # Job + run state are queried over the Scanner WebService REST surface:
#   GET /api/v1/scans/{scanId}            (status)
#   GET /api/v1/scans/{scanId}/manifest   (result manifest, when complete)
# or inspect the scanner schema directly in Postgres.

    Look for: a pattern in image types or sizes among the failing scans.

  3. Inspect the worker configuration in effect. There is NO stella scanner config get command. Worker settings are bound from the Scanner:Worker configuration section (env / ../etc/scanner.yaml). Confirm the live values for:

    • Scanner:Worker:MaxConcurrentJobs (default 2)
    • Scanner:Worker:Queue:MaxAttempts (default 5)
    • scanner:Queue:VisibilityTimeoutSeconds (WebService, default 300)
    • scanner:Queue:LeaseHeartbeatSeconds (WebService, default 30)

Deep diagnosis

  1. Analyze image complexity. The top-level stella image inspect command is fully implemented — it pulls the real manifest via IOciImageInspector, resolves a multi-arch index to platform manifests, and lists per-layer sizes and digests (ImageCommandGroup.BuildImageCommand → CommandHandlers.HandleInspectImageAsync in src/Cli/StellaOps.Cli/Commands/CommandHandlers.Image.cs). The image reference is a positional argument (not a --ref option):

    # Real manifest pull; --print-layers lists each layer's size + digest:
stella image inspect <image-ref> --print-layers --output json
# Other flags: --resolve-index (default true), --platform linux/amd64,
#              --timeout 60 (seconds). Requires network (blocked in offline mode).

    Watch for very large or very many layers / files. (A second, stubbed image surface — stella scan image inspect|layers --ref ... with hardcoded sha256:abc123... output, CommandFactory.cs BuildScanImageCommand — also exists; do not use it for diagnosis, its output is placeholder.) For result data you can also use the scan manifest (GET /api/v1/scans/{scanId}/manifest).

  2. Check worker concurrency / load. There is NO stella scanner workers stats command. The worker exposes only:

    stella scanner workers get          # shows configured worker count

    Live concurrency is bounded by Scanner:Worker:MaxConcurrentJobs. For runtime utilization, use the OTLP metrics (see Verification) or stella doctor --check check.scanner.resources.

  3. Profile a scan. There is NO stella scan image --image ... --profile command. The real ad-hoc scan entrypoint is stella scan run, which executes a scanner bundle against a directory target (not an image ref):

    stella scan run --runner docker --entry <scanner-image> --target <dir> --verbose

    To see which stage is slow, read the per-stage duration metric scanner_worker_stage_duration_ms (tagged by stage) rather than a CLI profiler flag.

  4. Check for filesystem-heavy images. There is no top-level stella image layers command and no --sort-by option anywhere. The top-level stella image group exposes only inspect. Use inspect with --print-layers to get real per-layer sizes (sort the JSON yourself):

    stella image inspect <image-ref> --print-layers --output json

    A single layer with a very large uncompressed footprint can trip the Pipeline.MaxExtractedBytes guard (default 8 GiB), which aborts extraction.

Resolution

Read first: Most “knobs” the previous version of this runbook listed (timeouts.scan, scan.incremental_mode, cache.layer_dedup, cache.sbom_cache, sbom.streaming_threshold, sbom.file_sample_max, vuln.parallel_matching, vuln.match_workers) and the commands that set them (stella scanner config set, stella scanner workers restart, stella db optimize) do not exist in the codebase. The real, source-backed levers are below.

Immediate mitigation

  1. Give a long job more time before its lease expires. Raise the queue visibility timeout so a slow scan is not redelivered mid-flight, and confirm the lease heartbeat is healthy:

    # scanner WebService config (section: scanner)
scanner:
  Queue:
    VisibilityTimeoutSeconds: 1200   # default 300
    LeaseHeartbeatSeconds: 30        # default 30
    MaxDeliveryAttempts: 5           # default 5

    Apply by updating the service configuration and restarting scanner-web / scanner-worker (no in-process “restart workers” CLI exists).

  2. Reduce per-worker contention. If all worker slots are busy, lower concurrency per worker or run more worker replicas:

    # scanner-worker config (section: Scanner:Worker)
Scanner:
  Worker:
    MaxConcurrentJobs: 1   # default 2 — fewer concurrent jobs, more headroom each

  3. Raise the extraction guard for legitimately large images. If a real (non-malicious) image exceeds the decompression-bomb cap:

    Scanner:
  Worker:
    Pipeline:
      MaxExtractedBytes: 17179869184   # 16 GiB (default 8 GiB = 8589934592)

Root cause fix

If the SBOM->matcher forward to Concelier is timing out:

The worker forwards component PURLs to Concelier’s learn endpoint with a per-request budget (Scanner:Worker:Pipeline:Matcher:TimeoutSeconds, default 60). Raise it if Concelier is slow under load, or disable the forward to isolate the scan:

Scanner:
  Worker:
    Pipeline:
      Matcher:
        TimeoutSeconds: 120   # default 60
        Enabled: true         # set false to skip the forward entirely

If the .NET reachability restore pre-step is timing out:

The optional dotnet restore pre-step (only when Scanner:Worker:Reachability:DotNetExtractor is msbuild/restore/M1) is bounded by DotNetRestoreTimeout (default 5m). For large solutions, raise the budget and/or supply an offline package source:

Scanner:
  Worker:
    Reachability:
      DotNetExtractor: auto              # `auto` skips the restore pre-step entirely
      DotNetRestoreTimeout: "00:10:00"   # default 00:05:00

For air-gap restore, set NUGET_OFFLINE_PACKAGES (see ADR-013 §M1).

If verdict push to an OCI registry is timing out:

Verdict push (when Scanner:Worker:VerdictPush:Enabled is true) has its own timeout and retry budget:

Scanner:
  Worker:
    VerdictPush:
      Timeout: "00:10:00"   # default 00:05:00
      MaxRetries: 3         # default 3

Verification

# Re-run the scanner Doctor checks and confirm no failures:
stella doctor --check check.scanner.queue
stella doctor --check check.scanner.resources

# Re-run an ad-hoc scan (directory target via a scanner bundle):
stella scan run --runner docker --entry <scanner-image> --target <dir> --verbose

Confirm the real OTLP metrics (Meter StellaOps.Scanner.Worker, see ScannerWorkerMetrics.cs) are healthy:

There is no scanner_scan_timeout_total metric to watch.

Prevention

Decisions & Risks