Runbook: Release Orchestrator - Resource Quota Exhausted

For platform and release on-call. Read this before treating any “quota exceeded” signal as the cause of a blocked release in Stella Ops. The short version: there is no live promotion-quota enforcement today — the quota surface that exists is an ingest source/tenant resource-consumption model whose management backend is not yet wired. If a release is blocked, the cause is a gate, workflow, or dependency, not a quota; this runbook shows you how to confirm that and where to go next.

Sprint: SPRINT_20260117_029_DOCS_runbook_coverage Task: RUN-004 - Release Orchestrator Runbooks

RECONCILIATION NOTICE (2026-05-30): This runbook was reconciled against source (src/ReleaseOrchestrator, src/JobEngine, src/Cli, and the scope catalog StellaOpsScopes.cs). Many original commands and symptoms described a promotion rate-limiting feature that does not exist in code. The quota surface that is defined is an ingest source/tenant resource-consumption quota (api_calls, data_ingested_bytes, items_processed, backfills, concurrent_jobs, storage_bytes), and even that surface is only partially wired: the CLI commands exist, but the orchestrator-side management backend is NOT IMPLEMENTED (the /api/v1/jobengine/quotas write paths return 501 Not Implemented from the release-orchestrator host, and a Platform compatibility layer answers the GET path with hardcoded demo data — values are mock, though this particular route does not carry an isDemo flag). Sections below are annotated INCORRECT / MISSING / NOT IMPLEMENTED accordingly. Treat the “roadmap” callouts as forward-looking, not operational.

Metadata

FieldValue
ComponentRelease Orchestrator / JobEngine (Scheduler)
SeverityMedium
On-call scopePlatform team, Release team
Last updated2026-05-30
Doctor checkNone ground-truthed for quotas. The real release Doctor checks (plugin StellaOps.Doctor.Plugin.Release) are check.release.active, check.release.promotion.gates, check.release.environment.readiness, check.release.rollback.readiness, and check.release.environment.matches-approved (run with stella doctor run --check <id>). The previously documented check.orchestrator.quota-status does not exist in source.
Required scopeorch:quota (StellaOpsScopes.OrchQuota) for quota management; orch:read (OrchRead) for read-only views

Reality check: what exists vs. what does not

Before triaging, understand the actual state of the quota subsystem in code.

Defined and wired (CLI layer):

NOT IMPLEMENTED (backend / enforcement):

Practical consequence: the CLI client (JobEngineClient) builds its URLs against {backendUrl}/api/v1/jobengine, so:

  • get → GET /api/v1/jobengine/quotas (with tenant/source_id/resource_type query params),
  • set → POST /api/v1/jobengine/quotas,
  • reset → POST /api/v1/jobengine/quotas:reset.

If you run stella orch quotas get, you will most likely receive the Platform mock payload (which does not match the CLI’s expected quotas[] shape, so the CLI prints “No quotas configured.” when result.Quotas.Count == 0). stella orch quotas set / reset POST to paths with no real handler (the Platform shim is GET-only; the orchestrator host answers 501) and fail. Do not rely on these as a live operational control until the JobEngine quota backend is implemented.

Symptoms

INCORRECT (original): The original symptoms (“promotions failing with quota exceeded”, OrchestratorQuotaExceeded alert firing, “daily quota exhausted”, “new promotions rejected”) describe a promotion rate-limiter that is not implemented. If you see promotions failing, the cause is almost certainly a gate, workflow, or dependency issue — see orchestrator-promotion-stuck.md, orchestrator-gate-timeout.md — not a quota.

What you may actually observe with the quota subsystem as it stands:

Impact

Impact TypeDescription
User-facingNone today: there is no live quota enforcement that blocks promotions or ingest.
Data integrityNo data loss. Quota counters/evidence are observational.
SLA impactNone attributable to quotas. If releases are blocked, look at gates/workflows, not quotas.

Diagnosis

Quick checks

  1. Confirm whether the quota path is even backed by a real service:

    # Expect demo data (Platform shim) or a 501 (orchestrator host).
stella orch quotas get --json
    • Demo payload (concurrentJobs, dailyJobLimit, no quotas[]) ⇒ you are hitting the Platform compatibility shim; the value is not authoritative.
    • 501 not_implemented ⇒ you are hitting the release-orchestrator host; quota management is unmigrated.

  2. Filter by resource type (real CLI flags):

    stella orch quotas get --tenant <tenant> --resource-type concurrent_jobs
# Valid resource types: api_calls, data_ingested_bytes, items_processed,
#                       backfills, concurrent_jobs, storage_bytes

  3. If a promotion is actually failing, run the real release Doctor checks:

    stella doctor run --check check.release.active
stella doctor run --check check.release.promotion.gates

Deep diagnosis

NOT IMPLEMENTED (original): stella promotion list --last 24h --count, ... --by-environment, ... --by-actor, stella orch quota reset-time, stella orch quota analyze — none of these exist. stella promotion only has assemble, attest, and verify subcommands (attestation tooling, not a promotion queue/history surface). stella orch only has sources, backfill, and quotas.

  1. Inspect quota evidence on a specific job run (read-only, the real surface): The JobEngine job detail (GET /api/v1/jobengine/jobs/{jobId}/detail on the Scheduler web service) returns a quotaEvidence block sourced from scheduler.runs.reason.quotaProof. Use it to confirm whether a quota decision was recorded for a run.

  2. Check ingest source health (sources, not promotions, are the consumers of these resource quotas):

    stella orch sources list --tenant <tenant>

Resolution

NOT IMPLEMENTED (original): The original “immediate mitigation” and “root cause fix” steps assumed live promotion quotas and commands that do not exist: stella orch quota request-increase, stella promotion priority set, stella promotion cancel, stella promotion create --test --dry-run, stella orch quota limits set, stella auth rate-limit set, stella orch config set ..., stella orch quota analyze. None of these are registered commands. They are retained below only as a description of the intended future capability (roadmap), clearly marked.

If a promotion is genuinely blocked (today’s real path)

Quotas are not the cause. Triage the actual orchestrator failure:

  1. Check active releases and gate state:
    stella doctor run --check check.release.active
stella doctor run --check check.release.promotion.gates
  2. Follow the relevant runbook:
    • Stuck promotion → orchestrator-promotion-stuck.md
    • Gate timeout → orchestrator-gate-timeout.md
    • Rollback failure → orchestrator-rollback-failed.md
    • Missing evidence → orchestrator-evidence-missing.md

Setting a resource quota (when the backend is wired)

The CLI is in place; the backend is not. Once the JobEngine quota backend lands, these are the real commands (note: quotas, plural — there is no stella orch quota status/limits form):

# Set a limit (requires orch:quota scope; --tenant, --resource-type, --limit are required)
stella orch quotas set \
  --tenant <tenant> \
  --resource-type concurrent_jobs \
  --limit 50 \
  --period daily \
  --warning-threshold 0.8

# Reset a usage counter (audited; --reason recommended)
stella orch quotas reset \
  --tenant <tenant> \
  --resource-type api_calls \
  --reason "Counter reset after false-positive spike"

Roadmap (NOT IMPLEMENTED — forward-looking only)

The following capabilities are described in older drafts but are not present in source. Do not attempt to run them:

When the JobEngine quota backend (StellaOps.Scheduler.WebService quota management endpoints) is migrated, this section should be reconciled against the shipped command surface and the orch:quota enforcement path.

Verification

# Confirm what the quota path returns (demo shim vs. 501 vs. real data)
stella orch quotas get --json

# Confirm the release pipeline itself is healthy (the thing that actually blocks releases)
stella doctor run --check check.release.active

INCORRECT (original): stella orch quota status, stella promotion list --status in_progress, stella promotion create --test --dry-run, and stella orch logs --filter "quota" ... are not real commands. There is no stella orch logs subcommand and no stella orch quota (singular) group.

Prevention

INCORRECT (original): docs/modules/release-jobengine/quotas.md and docs/operations/quota-management.md were referenced but do not exist. The module directory is docs/modules/jobengine/ (and docs/modules/release-orchestrator/), and there is no quota-management.md.