Runbook: Release Orchestrator - Required Evidence Not Found

Audience: Platform and Security on-call engineers responding to a Stella Ops release promotion blocked by the Security Gate for missing evidence (SBOM, scan, or reproducibility/attestation). Use this runbook to read the gate’s violation codes, locate the missing evidence, and produce it in the owning service so the promotion can re-evaluate.

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-004 - Release Orchestrator Runbooks

Reconciliation note (verify against src/ReleaseOrchestrator): This runbook was reconciled against the implemented Release Orchestrator. “Required evidence not found” is not a free-standing error — it is what the blocking Security Gate (StellaOps.ReleaseOrchestrator.Promotion.Gate.Security.SecurityGate, GateName = "security-gate", IsBlocking = true) reports when a promotion’s components are missing SBOMs, scans, or reproducibility/attestation evidence. The gate emits structured violation codes (e.g. SEC_SBOM_MISSING, SEC_SCAN_MISSING, SEC_SCAN_TOO_OLD, SEC_REPRO_EVIDENCE_MISSING, SEC_REPRO_DSSE_PROVENANCE_MISSING, SEC_REPRO_REKOR_UNVERIFIED, SEC_REPRO_EVIDENCE_SCORE_MISSING) that surface in the promotion’s Promotion.GateResults. Evidence inputs come from ISbomService / SbomRequirementChecker.HasSbomAsync(digest), IScannerService.GetLatestScanAsync(digest) (scan + ReproducibilityEvidence), and IEvidenceScoreService (Evidence Locker). Earlier drafts referenced a check.orchestrator.evidence-availability Doctor check, an OrchestratorEvidenceMissing alert, and a large CLI surface (stella promote {start|status|approve|reject|list}, stella promotion evidence/retry/status, stella evidence list/chain/store health/retry-store/policy show, stella scanner jobs/queue list, stella scanner config set, stella scan image --sbom-only/--force, stella attest build/verify-offline/predicates/create/status) — none of those exist in the shipped CLI. (A PromoteCommandHandler + BuildPromoteCommand() for stella promote … live in src/Cli/StellaOps.Cli/CliApplication.cs, but that command tree is not wired into the program entry point: Program.cs builds the root via CommandFactory.Create(...), which registers no top-level promote.) They have been corrected to the implemented surface or flagged NOT IMPLEMENTED below.

Metadata

FieldValue
ComponentRelease Orchestrator
SeverityHigh
On-call scopePlatform team, Security team
Last updated2026-05-30
Doctor checkcheck.release.promotion.gates (Release Pipeline plugin; default severity Warn)

The Doctor check above is the closest real diagnostic — PromotionGateHealthCheck (src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Release/Checks/PromotionGateHealthCheck.cs). It queries pending approvals from GET /api/v1/release-orchestrator/approvals?statuses=pending and evaluates their persisted gateResults, including denied, advisory, skipped, exception, and unevaluated outcomes. There is no check.orchestrator.evidence-availability check in source. The full Release Pipeline plugin ships check.release.active, check.release.promotion.gates, check.release.rollback.readiness, check.release.environment.readiness, and check.release.environment.matches-approved — there is no check.orchestrator.* family.

Symptoms

NOT IMPLEMENTED: there is no OrchestratorEvidenceMissing alert in source. The evidence-readiness signal is the per-component evidence check inside SecurityGate (its violation codes) plus the Doctor check.release.promotion.gates result. Verify any dashboard alert is wired to those, not to a nonexistent metric.

Impact

Impact TypeDescription
User-facingPromotion blocked by the blocking Security Gate until evidence is generated
Data integrityArtifact is safe; the gate is fail-closed by design. A missing SBOM/scan/attestation is the correct blocking outcome, not data loss
SLA impactRelease blocked; security/compliance requirements not met

Diagnosis

Quick checks

  1. Run the Doctor promotion-gate health check:

    stella doctor --check check.release.promotion.gates

    This verifies required policies are loaded, the attestor is reachable, and approvers are configured. (--check takes a single check ID — there are no wildcards.)

  2. Inspect the blocked promotion’s status and gate results via the API. A failed security-gate with violation codes (SEC_SBOM_MISSING, SEC_SCAN_MISSING, etc.) on the promotion’s gate results is the “required evidence not found” condition. The v2 approval API surfaces the gate trace:

    # GET /api/v1/approvals/{id}/gates  → { approvalId, decisionDigest, gates[] }
curl -s "$ORCH_URL/api/v1/approvals/<approval-id>/gates" \
  -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT"

    Requires release:read (ReleaseControlV2Endpoints.cs, MapGroup("/api/v1/approvals")).

    NOT IMPLEMENTED: there is no stella promote status <promotion-id> command in the shipped CLI. A PromoteCommandHandler (StartAsync/StatusAsync/ApproveAsync/ RejectAsync/ListAsync against /api/v1/promotions) and a matching BuildPromoteCommand() exist only in src/Cli/StellaOps.Cli/CliApplication.cs, which is not wired to the program entry point — Program.cs builds the root command via CommandFactory.Create(...), and that root registers no top-level promote command. The only promote verbs in the real CLI are stella policy promote <policy-id> --from --to (env-to-env policy promotion, no lifecycle/status) and stella orch (the Source & Job Orchestrator, unrelated to release promotion). Inspect promotion/approval state through the orchestrator API or Console, not a promote CLI.

  3. List evidence packets registered for the release (Release Orchestrator evidence API):

    # GET /api/release-orchestrator/evidence?releaseId=<release-id>   (also /api/v1/release-orchestrator/evidence)
curl -s "$ORCH_URL/api/release-orchestrator/evidence?releaseId=<release-id>" \
  -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT"

    Requires the release:read scope. type and releaseId are the supported filters; each row carries id, releaseId, type, hash, algorithm, and imageDigest.

Deep diagnosis

  1. Read the Security Gate violation codes. The blocking security-gate (…/Promotion/Gate/Security/SecurityGate.cs) checks each release component and emits a policyViolationCodes array in the gate details. Map the code to the missing evidence:

    • SEC_SBOM_MISSING — component has no SBOM (SbomRequirementChecker.HasSbomAsync → ISbomService.GetByDigestAsync returned null). Only emitted when requireSbom is on (default true).
    • SEC_SCAN_MISSING — component has no security scan (IScannerService.GetLatestScanAsync returned null). Also gated by requireSbom — with requireSbom=false a missing scan does not raise this code (the gate skips the component’s scan-derived checks).
    • SEC_SCAN_TOO_OLD — scan age exceeds maxScanAge (default 24h).
    • SEC_REPRO_EVIDENCE_MISSING — gate requires reproducibility/attestation evidence but none is attached.
    • SEC_REPRO_DSSE_PROVENANCE_MISSING / SEC_REPRO_DSSE_INTOTO_MISSING / SEC_REPRO_REKOR_UNVERIFIED / SEC_REPRO_EVIDENCE_SCORE_MISSING — specific required evidence facets absent (only enforced when the matching require* config flag is set; all default to false).

  2. Confirm which evidence the gate config requires. The gate is permissive by default — only requireSbom (default true) is on; all requireDsse*, requireRekorVerification, requireEvidenceScoreMatch, etc. default to false. If a SEC_REPRO_* violation fires, the environment’s gate config turned that requirement on. Review the environment gate configuration (EnvironmentGateConfig / SecurityGateConfig) before generating evidence.

  3. Check the approval’s structured evidence reference set (v2 approval API):

    # GET /api/v1/approvals/{id}/evidence  → { packet, manifestDigest, decisionDigest }
curl -s "$ORCH_URL/api/v1/approvals/<approval-id>/evidence" \
  -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT"

    Used to verify the SBOM/attestation/scan reference completeness for an approval decision.

  4. Verify the evidence packet integrity (Release Orchestrator evidence API):

    # POST /api/release-orchestrator/evidence/{id}/verify  — recomputes and compares the sha256 content hash
curl -s -X POST "$ORCH_URL/api/release-orchestrator/evidence/<evidence-id>/verify" \
  -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT"

NOT IMPLEMENTED: there is no stella promotion evidence <id> --missing, no stella evidence list/chain, no stella scanner jobs/queue list, and no stella attest status. Verified against src/Cli/StellaOps.Cli/Commands/CommandFactory.cs (the real CLI root):

  • stella promotion builds promotion attestations — subcommands are assemble, attest, and verify (stella.ops/promotion@v1), not lifecycle/evidence queries. (detscore is a separate top-level command, stella detscore run, not a promotion subcommand.)
  • stella evidence subcommands are export, verify, store, status, card, reindex, verify-continuity, migrate, holds, audit, replay, proof, provenance, seal, push-referrer, and list-referrers — there is no evidence list or evidence chain.
  • stella attest subcommands are sign, verify, list, show, fetch, key, bundle, attach, oci-list, oci-verify, and link — there is no build, verify-offline, create, or predicates subcommand. (attest verify already verifies a DSSE envelope offline against policy/trust roots; attest list filters by --subject/--type/--issuer/ --tenant/--scope, while --image belongs to attach/oci-list/oci-verify.)

Resolution

There is no orchestrator-side “regenerate evidence” command. The orchestrator consumes evidence produced by Scanner (SBOM + scan), the Attestor/Signer (attestations), and the Evidence Locker (evidence score). Resolve a SEC_* violation by producing the missing evidence in the owning service for the component digest, then re-evaluating the gate.

Immediate mitigation

  1. Scan the component image to produce SBOM + scan evidence. Use the implemented scan surface (stella scan run … / stella scan upload …, src/Cli/StellaOps.Cli/Commands/CommandFactory.cs) — there is no stella scan image --sbom-only/--force. stella scan image only has inspect/layers. The Security Gate reads the latest scan for the component digest, so a fresh successful scan clears SEC_SCAN_MISSING/SEC_SCAN_TOO_OLD.

  2. Produce missing attestations (only needed when the gate config requires them). Sign and attach via the attestor surface:

    stella attest sign --envelope provenance.json ...   # create + sign a DSSE attestation
stella attest attach --image <image-ref> ...        # attach the attestation to the OCI artifact
stella attest list --subject <image-digest>         # confirm the attestation is recorded

    (stella attest subcommands are sign/verify/list/show/fetch/key/bundle/attach/oci-list/ oci-verify/link; there is no attest build or attest create.) See attestor-signing-failed.md if signing itself fails.

  3. Register the evidence packet with the orchestrator so promotion/deployment gates can rely on durable evidence truth:

    # POST /api/release-orchestrator/evidence  (requires release:write)
# body: { releaseId, type, rawContentBase64, algorithm? } — the service recomputes the sha256 hash

Root cause fix

If the scan never ran for the component:

  1. The gate evaluates the component digest, so confirm the release component digest matches a scanned artifact. Components are registered via the release/component API (/api/release-orchestrator/..., release:write).
  2. Re-run the scan against that exact digest (stella scan run …), then re-evaluate.

If evidence exists but the gate config requires more than is present:

  1. A SEC_REPRO_* violation means a require* flag is on in the environment’s SecurityGateConfig (e.g. requireDsseProvenance, requireRekorVerification, requireEvidenceScoreMatch, minEvidenceScore). Either produce the required evidence facet, or correct the gate configuration if the requirement was set in error.

If attestation signing failed:

  1. See attestor-signing-failed.md and attestor-rekor-unavailable.md. The Doctor promotion-gate check pings {Attestor:Url}/health; an unreachable attestor surfaces as a check.release.promotion.gates Warn.

If the evidence score is not ready (Evidence Locker):

  1. SEC_REPRO_EVIDENCE_SCORE_MISSING / SEC_REPRO_EVIDENCE_SCORE_NOT_READY means IEvidenceScoreService.GetScoreAsync returned null or a non-ready status. If asyncHoldSlaHours is configured the gate emits a non-blocking hold_async outcome (SEC_HOLD_ASYNC_EVIDENCE_NOT_READY) instead of a hard fail; wait for the score to become ready or escalate per the SLA. See evidence-locker-ops.md.

Verification

# Confirm the evidence packet is registered and integrity-verified
curl -s -X POST "$ORCH_URL/api/release-orchestrator/evidence/<evidence-id>/verify" \
  -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT"

# Re-evaluate the promotion: re-check the gate trace via the v2 approval API
# (there is no `promote`/`promotion` lifecycle CLI and no `promotion retry`)
curl -s "$ORCH_URL/api/v1/approvals/<approval-id>/gates" \
  -H "Authorization: Bearer $TOKEN" -H "X-Tenant-Id: $TENANT"

# Re-run the Doctor gate health check
stella doctor --check check.release.promotion.gates

NOT IMPLEMENTED: stella promote status, stella promotion retry, stella promotion status, stella scanner queue list, stella scanner config set auto_scan.*, stella evidence store health, stella evidence retry-store, and stella evidence policy show do not exist. (The orchestrator promotion lifecycle has no dedicated CLI at all in the shipped CommandFactory root — see the Diagnosis note above.) Auto-scan triggers and evidence retention are not configured through the CLI surface documented here — verify the relevant service config (Scanner, Evidence Locker) directly. A Failed promotion re-enters the lifecycle via RolledBack or AwaitingApproval (PromotionStateMachine), not a retry verb.

Prevention

NOT IN REORG TREE: docs/operations/evidence-requirements.md (referenced by earlier drafts) does not exist under docs/operations/. Use the evidence-locker dossier (docs/modules/evidence-locker/architecture.md) and the promotion-evidence contract above instead.