Runbook: Scanner — SBOM Generation Failures

Use this runbook when scans complete but SBOM generation, composition, or validation fails — leaving partial or missing components, schema-validation errors, or a degraded check.scanner.sbom success rate. It is written for the Platform on-call team. Because reachability analysis and SBOM-gated release promotions cannot proceed without a valid SBOM, treat these failures as release-blocking.

Stella Ops generates and validates SBOMs in two formats only — CycloneDX and SPDX. For related symptoms, see scanner-oom.mdand scanner-timeout.md.

Metadata

FieldValue
ComponentScanner
SeverityHigh
On-call scopePlatform team
Last updated2026-05-30
Doctor checkcheck.scanner.sbom

Reconciled against src/Scanner and the Doctor SBOM health check on 2026-05-30. The Doctor check ID is check.scanner.sbom (defined in src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Scanner/Checks/SbomGenerationHealthCheck.cs). Several CLI commands and config keys that earlier drafts of this runbook referenced are not implemented and have been removed or flagged inline.

Symptoms

NOTE (NOT IMPLEMENTED): there is no ScannerSbomGenerationFailed Prometheus alert and no scanner_sbom_generation_failures_total metric in the codebase (src/ and devops/ contain neither). Health is surfaced through the Doctor check above, not a dedicated alert/counter. If a Prometheus alert is required, track it as a follow-up rather than relying on it here.

Impact

Impact TypeDescription
User-facingIncomplete vulnerability coverage; missing dependencies not scanned
Data integrityPartial SBOM may miss vulnerabilities; attestations incomplete
PipelineReachability analysis and SBOM-gated release promotions cannot proceed without a valid SBOM
SLA impactSBOM generation success rate degraded (Doctor warns < 95%, fails < 80%)

Diagnosis

Quick checks

  1. Run the Doctor SBOM health check:

    stella doctor --check check.scanner.sbom
# or, explicitly via the run sub-command:
stella doctor run --check check.scanner.sbom

    This queries the Scanner service at /api/v1/sbom/stats and reports total_generated, successful_generations, failed_generations, success_rate, format_cyclonedx, format_spdx, and validation_failures. The check needs Scanner:Url (or Services:Scanner:Url) configured.

    NOTE: the Doctor check depends on the Scanner exposing /api/v1/sbom/stats. That endpoint is not currently mapped in StellaOps.Scanner.WebService (only /api/v1/sbom/validate, /api/v1/sbom/validators, /api/v1/sbom/upload, and /api/v1/sbom/{scanId}/evidence exist). If the check reports “Cannot retrieve SBOM stats”, treat the stats endpoint as a known gap and fall back to scanner logs below.

  2. Inspect scanner service logs for SBOM/composition failures:

    docker compose -f devops/compose/docker-compose.stella-ops.yml logs scanner | grep -iE "sbom|composition|validation"

    Look for: composition/validation errors, unsupported format messages, out-of-memory or timeout signals during emit.

  3. Check the scan status for the affected scan:

    # GET /api/scans/{scanId} — returns status + failureReason
curl -sS -H "Authorization: Bearer $TOKEN" \
  https://stella-ops.local/api/scans/<scan-id> | jq '{status, failureReason}'

    The failureReason field carries the recorded failure for the scan.

Deep diagnosis

  1. Validate a specific SBOM document against the schemas:

    # POST /api/v1/sbom/validate — validates against CycloneDX or SPDX schema
curl -sS -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/vnd.cyclonedx+json" \
  --data-binary @sbom.cdx.json \
  "https://stella-ops.local/api/v1/sbom/validate"

    The response is an SbomValidationResponseDto: check isValid, errorCount / warningCount, and the diagnostics[] array — each diagnostic carries severity, code, message, path, line, and suggestion describing the schema/format problem. (There is no top-level errors[] field on this endpoint; that key only appears on the BYOS upload’s 400 problem-details.) Requires the scanner.scans.read policy (scope scanner:read).

    NOTE: if the validation gate is disabled (Enabled = false or Mode = Off), this endpoint short-circuits and returns isValid: true with validatorName: "validation-disabled" regardless of the document — so a “valid” result here does not prove the SBOM is well-formed. A 503 with validatorName/diagnostic code VALIDATOR_UNAVAILABLE means the validator itself could not run.

  2. List available SBOM validators / supported formats:

    # GET /api/v1/sbom/validators
curl -sS -H "Authorization: Bearer $TOKEN" \
  "https://stella-ops.local/api/v1/sbom/validators"

    Stella generates and validates two SBOM formats: CycloneDX and SPDX (SbomFormats.CycloneDx / SbomFormats.Spdx). There is no third “generator” backend.

  3. Confirm language-analyzer plug-ins load and verify:

    # Enumerates analyzer plug-ins under a directory and checks signatures.
stella scanner plugins list <plugin-dir>
stella scanner plugins verify --manifest <manifest-path>

    Problem if: an analyzer plug-in fails signature verification or is missing, so the corresponding ecosystem is not analyzed. (These commands operate on plug-in directories/manifests; there is no per-plugin enable/restart/log surface in the CLI.)

Resolution

Immediate mitigation

  1. Restart the scanner to clear bad cached state:

    docker compose -f devops/compose/docker-compose.stella-ops.yml restart scanner

  2. Re-run the scan once the underlying cause is addressed:

    # POST /api/scans — resubmit the image (use force to bypass cached results)
curl -sS -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"image":{"reference":"<image-ref>"},"force":true}' \
  "https://stella-ops.local/api/scans"

    Requires the scanner.api policy (scope scanner:scan).

  3. Supply a Bring-Your-Own-SBOM (BYOS) document for the scan when the built-in composition cannot complete and you have a trusted SBOM:

    # POST /api/scans/{scanId}/sbom — accepts CycloneDX 1.6/1.7 or SPDX JSON
curl -sS -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/vnd.cyclonedx+json; version=1.7" \
  --data-binary @sbom.cdx.json \
  "https://stella-ops.local/api/scans/<scan-id>/sbom"

    Requires the scanner.scans.write policy (scope scanner:write). Accepted content types: application/vnd.cyclonedx+json (version 1.6/1.7), application/spdx+json, application/json.

    NOTE: a host-wide --sbom-fallback / --sbom-partial-ok flag and an sbom.fallback_mode config key do not exist. BYOS upload above is the supported way to substitute an SBOM.

Root cause fix

If an ecosystem is not analyzed (missing/failed analyzer plug-in):

  1. Enumerate and verify the language-analyzer plug-ins:

    stella scanner plugins list <plugin-dir>
stella scanner plugins verify --manifest <manifest-path>

  2. Re-deploy the correct (signed) analyzer plug-in bundle and restart the scanner so the plug-in loader picks it up. Supported ecosystems include the OS package managers (apk, dpkg, rpm, pacman, portage, homebrew, etc.) and the language analyzers under src/Scanner/StellaOps.Scanner.Analyzers.Lang.*.

    NOTE: the CLI has no stella scanner plugins enable/restart/logs and no --type package-manager filter, and there is no sbom.custom_mappings.<format> config key. Analyzer coverage is managed by deploying signed plug-in bundles, not by per-plugin CLI toggles.

If SBOM schema validation is failing:

  1. Validate the document to see the precise schema error (see Deep diagnosis step 1). The emit-side validation pipeline (StellaOps.Scanner.Emit.Composition.SbomValidationPipeline) validates both CycloneDX and SPDX output; FailOnError (default true) means a failed validation aborts composition.
  2. Correct the source artifact (truncated layers, malformed/missing manifests) or upgrade the scanner image so the generator and schema version match, then re-run the scan.

If source artifacts are corrupted:

  1. Re-pull or rebuild the image and confirm the digest matches what is being scanned (deployments pull by digest, not tag).
  2. Report unrecoverable layers to the image owner for a rebuild.

If memory exhaustion or timeout occurs during composition:

  1. Increase the scanner container memory and watch for OOM:
    # devops/compose/docker-compose.stella-ops.yml
services:
  scanner:
    deploy:
      resources:
        limits:
          memory: 4G

    docker compose -f devops/compose/docker-compose.stella-ops.yml up -d scanner
  2. The SBOM validation pipeline has a default ValidationTimeout of 60s (SbomValidationPipelineOptions.ValidationTimeout). Very large images may exceed it; raise scanner resources or split the workload rather than relying on a per-command timeout flag (no sbom.timeout / sbom.streaming_mode config key exists).

Verification

# Re-run the Doctor SBOM health check
stella doctor run --check check.scanner.sbom

# Re-validate the produced SBOM against the schema
curl -sS -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/vnd.cyclonedx+json" \
  --data-binary @sbom.cdx.json \
  "https://stella-ops.local/api/v1/sbom/validate"

# Confirm the scan completed
curl -sS -H "Authorization: Bearer $TOKEN" \
  https://stella-ops.local/api/scans/<scan-id> | jq '{status, failureReason}'

Prevention