Runbook: Release Orchestrator - Gate Evaluation Timeout

Audience: Platform on-call engineers responding to Stella Ops release promotions stalled or denied because the CVE-aware policy-engine gate timed out (or failed closed). Use this runbook to tune the policy-gate timeout, diagnose policy-engine reachability, and safely approve or fail-open a stuck gate.

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-004 - Release Orchestrator Runbooks

Reconciliation note (verify against src/ReleaseOrchestrator): This runbook was reconciled against the implemented Release Orchestrator. The single real “gate timeout” in code today is the per-request HTTP timeout on the CVE-aware policy-engine gate call (ReleaseOrchestrator:Policy:TimeoutSeconds, default 10s), implemented in HttpPolicyEngineGateClient. On timeout the client fails closed (returns a deny verdict) unless FailClosed is set to false. Earlier drafts referenced a generic per-gate timeout framework with stella orch/stella promotion sub-commands, a check.orchestrator.gate-timeout Doctor check, an OrchestratorGateTimeout alert, and an orchestrator_gate_timeout_total metric — none of those exist in source. They have been corrected to the implemented surface or flagged as NOT IMPLEMENTED below.

CLI caveat (updated 2026-07-03): the stubbed stella release gates sub-commands (list/status/approve/reject) were removed from the CLI — they printed fabricated sample output. stella release now carries only the real HTTP-backed operator verbs approve/reject/promote/deploy/rollback (ReleaseOperatorDecisionCommandGroup.cs). The gate-state surfaces are the Doctor check (check.release.promotion.gates) and the approvals REST API. Also: stella policy list (suggested by the Doctor remediation) is NOT an implemented stella policy sub-command — use it only as a pointer, not a runnable command.

Metadata

FieldValue
ComponentRelease Orchestrator
SeverityHigh
On-call scopePlatform team
Last updated2026-05-30
Doctor checkcheck.release.promotion.gates (Release Pipeline plugin; default severity Warn)

The Doctor check above is the real gate-health diagnostic (src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Release/Checks/PromotionGateHealthCheck.cs). It is not a dedicated timeout check — it reports gate configuration health (missing policies, unreachable attestor, missing approvers) and surfaces orchestrator/policy-engine reachability. There is no check.orchestrator.gate-timeout check in source.

Symptoms

NOT IMPLEMENTED: there is no OrchestratorGateTimeout alert and no orchestrator_gate_timeout_total metric in source. The gate metrics that are emitted by ReleaseOrchestratorPluginMonitor are stellaops_gate_evaluations_total and stellaops_gate_evaluation_duration_ms. The module dossier’s metrics spec (docs/modules/release-orchestrator/operations/metrics.md) additionally lists stella_gate_evaluations_total, stella_gate_evaluation_duration_seconds, and stella_gate_blocks_total — note the doc uses a stella_ prefix while the emitter uses stellaops_; verify the exposed names on your build before wiring alerts.

Impact

Impact TypeDescription
User-facingPromotions denied or stalled; release pipeline blocked until the gate succeeds or is overridden
Data integrityNo data loss; the promotion request is re-evaluable. Fail-closed deny is the safe default — a vulnerable release cannot slip through a temporary policy-engine outage
SLA impactRelease SLO violated if the policy-engine outage / slow evaluation persists

Diagnosis

Quick checks

  1. Run the Doctor gate-health check:

    stella doctor --check check.release.promotion.gates

    This queries the orchestrator and (if reachable) the policy engine + attestor. A Warn with connection_error_type=timeout or Cannot retrieve promotion gates points at an unreachable/slow orchestrator or policy engine.

  2. Inspect gate status for the release / configured gates for the environment.

    REMOVED (2026-07-03): the stubbed stella release gates status/list commands were deleted from the CLI (they printed canned sample tables, no HTTP call). Use the Doctor check (stella doctor --check check.release.promotion.gates), the orchestrator logs, the approvals REST API, or the UI Gate policies / promotion decision cockpit (/releases/promotions/<id>) for ground truth.

Note on neighbouring command groups: the stella orch group operates the Source & Job Orchestrator (sources/backfill/quotas) and has no gate sub-commands; stella promotion builds promotion attestations (assemble), not gate operations.

Deep diagnosis

  1. Read the orchestrator logs for the policy-gate call. The CVE-aware gate (HttpPolicyEngineGateClient) logs the following on the failure paths — grep the release-orchestrator container logs:

    • Policy.Engine evaluate timed out after {Timeout}s for pack {PackId}/{Version}.
    • Policy.Engine evaluate returned {Status} ... (non-2xx from the engine)
    • Policy.Engine evaluate network failure ... Each of these routes into the fail-closed (deny) branch when FailClosed=true.

  2. Confirm the resolved policy pack. The gate resolves the pack in this order (HttpPolicyEngineGateClient.ResolvePackAsync):

    1. DB binding — release_orchestrator.environment_policy_bindings (operator-managed)
    2. Static config map ReleaseOrchestrator:Policy:EnvironmentBindings (DEPRECATED back-compat)
    3. Tenant default ReleaseOrchestrator:Policy:DefaultPackId + DefaultPackVersion
    4. If none resolves, the policy gate is skipped and legacy approval-policy logic runs. The log line Resolved policy pack {PackId}@v{Ver} ... via {Source} shows which path won.

  3. Check policy-engine availability. The Doctor remediation for this check emits the literal string stella policy list as its suggested next step (PromotionGateHealthCheck.cs). NOTE: list is not an implemented sub-command of the stella policy group in source — the group exposes simulate, activate, lint, edit, test, new, history, explain, init, compile, version, submit, review, publish, rollback, sign, verify-signature, lattice, verdicts, and promote, but no list (and no stats). Treat stella policy list as Doctor remediation copy, not a runnable command on the current build; use a real policy sub-command (e.g. stella policy verdicts) or the policy-engine /health endpoint to confirm reachability. See policy-evaluation-slow.md for engine-side latency triage.

  4. Check attestor reachability if gates require attestations — the Doctor check probes the attestor /health endpoint and reports attestor_unavailable when it cannot reach it:

    stella doctor --check check.attestation.*

Resolution

Immediate mitigation

  1. Increase the policy-gate timeout by raising the per-request timeout used for the policy-engine call. This is the only real “gate timeout” knob:

    // appsettings / environment override, section "ReleaseOrchestrator:Policy"
"ReleaseOrchestrator": {
  "Policy": {
    "TimeoutSeconds": 30   // default is 10
  }
}

    Apply via the service’s configuration (env var ReleaseOrchestrator__Policy__TimeoutSeconds=30) and restart/redeploy the release-orchestrator service. There is no stella orch config set command and no per-gate gates.<name>.timeout key in source.

  2. Emergency fail-open (last resort). If the policy-engine is down and you must let promotions fall back to legacy approval-policy matching instead of denying, set fail-open. This intentionally weakens the CVE gate, so treat it as a temporary, audited rollback only:

    "ReleaseOrchestrator": { "Policy": { "FailClosed": false } }

    To bypass the CVE-aware gate entirely, set ReleaseOrchestrator:Policy:Enabled=false (master kill-switch).

  3. Override / approve a stuck gate. Manual approval and rejection are real, backend-wired operations on the approvals API, which require the release:publish scope (policy ReleaseOrchestratorPolicies.ReleaseApprove):

    • POST /api/release-orchestrator/approvals/{id}/approve (and /reject), also mirrored under /api/v1/release-orchestrator/approvals. The approve/reject routes are Audited (AuditActions.Release.ApproveRelease / RejectRelease).

    The former stubbed CLI shortcut (stella release gates approve) was removed on 2026-07-03 — during an incident drive the approval through the real surfaces:

    # Real CLI action (HTTP-backed, requires release:publish):
stella release approve <approval-id> --comment "Policy-engine outage - approved by <approver>"

# Or call the approvals API directly:
#   POST /api/release-orchestrator/approvals/{approval-id}/approve

    The dedicated scope to bypass release policy gates is release:bypass (StellaOpsScopes.ReleaseBypass, in the canonical catalog). Verify which surfaces honor it before relying on a “skip gate” flow — there is no stella promotion gate skip command in source. The CLI stella release promote --force advertises bypassing non-blocking approval gates (its sample output skips manual-approval) but is itself a stubbed/sample-output command and performs no backend promotion.

Root cause fix

If the policy engine is slow or returning errors:

  1. Triage engine-side latency with the policy-evaluation-slow.md runbook.
  2. Confirm the orchestrator is calling the engine with the required tenant header — the client attaches X-StellaOps-TenantId; without it the engine returns 400 POLICY_TENANT_HEADER_REQUIRED and the gate fails closed.
  3. Confirm the bypass-network/routing between release-orchestrator and policy-engine is intact — a misrouted call surfaces as 401 and routes into the fail-closed branch.

If the gate configuration itself is unhealthy (Doctor reports missing_policies, attestor_unavailable, or no_approvers):

  1. Verify required policy packs are bound to the environment via release_orchestrator.environment_policy_bindings (REST: EnvironmentPolicyBindingEndpoints, scopes release.policy-bindings.read / release.policy-bindings.write).
  2. Ensure the attestor service is reachable if gates require attestations.
  3. Ensure approvers are configured for any gate that requires manual approval.

NOT IMPLEMENTED in source (forward-looking / roadmap only — do not run): gate-level retry/backoff (gates.<name>.retries), gate result caching (gates.<name>.cache_ttl), per-gate circuit breakers (gates.<name>.circuit_breaker.*), evidence pre-fetching (gates.evidence_prefetch), evidence cache sizing (evidence.cache_*), and OPA worker tuning (policy config set opa.workers). A connector-level circuit breaker (ConnectorCircuitBreaker) exists for IntegrationHub connectors, but it is not a gate timeout control.

Verification

# Re-run the Doctor gate-health check. It reads persisted gateResults from the
# orchestrator's pending-approval API and reports blocking/advisory/unevaluated truth.
stella doctor --check check.release.promotion.gates

# The former `stella release gates status` command was removed 2026-07-03 (it printed
# stubbed sample output). Use the orchestrator logs / approvals API to confirm the fix.

The authoritative confirmation is the orchestrator log line on the next promotion: after raising TimeoutSeconds (or restoring the policy engine), re-submit the promotion and confirm the policy-gate verdict reads Policy pack {pack}@v{ver} returned allow ... (gate Outcome=approve, when the pack allows) rather than the fail-closed ... Fail-closed: treating as deny.

Prevention