Runbook — routine database maintenance

Audience: operators of a Stella Ops installation. Short version: there is nothing for you to do. Routine database maintenance — retention pruning and planner-statistics refresh — runs nightly through the product’s own scheduler on every installation. This page exists so you can see what it does, confirm it is running, and change it if you want to.

Owner ruling, 2026-08-10: Stella Ops must be self-serviceable. No installation may depend on an operator (or on us) running psql rituals or authorizing routine retention. If you find yourself hand-running SQL to keep a database from growing, that is a defect — please report it.

What runs, and when

01:00 is deliberately before the 02:00 nightly scan: reclaim first, then scan against a database that is not carrying the backlog.

Each service’s pass does two things, both bounded:

1. Retention pruning — reclaims event envelopes that every consumer has durably processed AND that are older than the service’s declared window. Nothing unconsumed is ever deleted; see the eventing contract in StellaOps.Eventing.Reliability.
2. ANALYZEof the hot tables, so the query planner’s statistics describe the table as it now is.

It never takes an exclusive lock. VACUUM FULL and CLUSTER take an ACCESS EXCLUSIVE lock on tables every consumer touches on every event; a nightly job must never be able to freeze the estate on its own initiative. If you ask for one (vacuum: true), the service answers vacuumPerformed: false and says why, rather than silently doing less than you asked.

Is it running?

The trigger loop says which mode it is in at startup, so “nothing fires” is never a mystery:

docker logs stellaops-scheduler-web 2>&1 | grep -i "cron triggering"

• Cron triggering enabled: polling every … — running.
• Cron triggering is DISABLED (Scheduler:Worker:Trigger:Enabled=false); stored schedules will NOT fire. — not running. Runs can still be created by hand through the API, but nothing is scheduled.

Then look at the runs themselves. A cron run is an ordinary run: it appears in GET /api/v1/scheduler/runs, it can be cancelled with POST /api/v1/scheduler/runs/{runId}/cancel, and it produces an audit event — exactly like a run you started yourself.

Turning it on

As of this release the loop ships DISABLED by default. That is a staging posture for the first supervised activation, not the intended end state: the ruling is nightly-by-default on every installation, and a later release flips it. Enabling it wakes every schedule that already exists on your estate, so do it deliberately and watch the first night.

# devops/compose/.env
SCHEDULER_TRIGGER_ENABLED=true

Then recreate the scheduler:

docker compose … up -d --force-recreate --no-deps scheduler-web

Changing it

Everything below is ordinary configuration; none of it needs a rebuild.

To change when maintenance runs, edit the schedule itself — it is a normal schedule:

PATCH /api/v1/scheduler/schedules/sys-default-nightly-db-maintenance
{ "cronExpression": "0 3 * * *" }

Your edit survives restarts. The system schedules are created once if absent and never overwritten.

There is no connection string here, and that is deliberate. The scheduler never opens a connection to any service’s database; it asks each service to maintain its own (CoC §8.2). A service that has not implemented the maintenance contract simply is not listed.

The misfire grace, in plain terms

If the scheduler is down when a job was due, should it run late or wait for the next slot? MisfireGrace is that answer, and it bounds catch-up to at most one run per schedule — never a backfill of everything missed.

• Down for 5 minutes across a 01:00 job, grace 10 minutes ⇒ it runs at start-up.
• Down for an hour ⇒ 01:00 is skipped and it runs tomorrow at 01:00.
• Down for a week ⇒ still just tomorrow at 01:00. It never replays the week.

Reading the result

The job logs one line per adopter, with numbers the owning service measured against its own database:

Maintenance completed for 'vulnerabilities': 50000 row(s) reclaimed, 41943040 byte(s) released.

Two things that look like problems and are not:

• batchLimitReached: true. Each pass is bounded so it cannot flood the WAL. A large first backlog drains over several nights. Watch the age of the oldest row rising, not only the row count falling.
• Bytes released is zero or negative. Deleting rows leaves dead tuples that only autovacuum returns to the free space map, and ingest keeps writing throughout. A pass that reclaimed 50,000 rows while the table grew is a real and honest signal. We report the raw before/after rather than clamping it to a flattering number.

Running it now, by hand

Not required — but if you want a pass immediately, trigger the schedule through the ordinary run API rather than touching the database:

POST /api/v1/scheduler/runs
{ "scheduleId": "sys-default-nightly-db-maintenance", "reason": { "trigger": "manual" } }

Or call one service’s maintenance directly (needs ops.health):

curl -sX POST -H "Authorization: Bearer $TOKEN" \
  https://stella-ops.local/api/vulnerabilities/v1/maintenance/runs \
  -H 'content-type: application/json' -d '{"reason":"manual"}'

If a run fails with an authorization or configuration error

A failed run records which service failed and why, for example (2026-09-30):

Scheduler job plugin 'database-maintenance' failed: Database maintenance failed for 1 of 1 service(s): vulnerabilities: Token request failed with status 400 (invalid_scope).

invalid_scope comes from Authority, not from the maintained service. The job authenticates through the JobEngine host’s shared token client stellaops-scheduler-concelier-observations (configured under Scheduler:Worker:Concelier:Authority) and asks for ops.health, the scope every adopter’s maintenance API requires. Authority refused the token because that client’s registration does not allow the scope.

The repair belongs to the Authority client registration of that one client: its allowed scopes must include ops.health. The shipped descriptor (devops/etc/authority/plugins/standard.yaml) grants it, and Authority reconciles the descriptor’s clients on start, adding missing scopes without removing any. Access > Clients in the Console shows each client’s allowed scopes. Add only ops.health to that client. Do not give it, or any operator, an administrator role or unrelated scopes: the refused token belongs to the service, not to a person.

The Scheduled Jobs page names this failure as an authorization or configuration failure with its repair owner and does not offer Retry, because a retry sends the same token request and is refused again. After the repair, start the schedule with Run Now (Scheduled Jobs > Schedules) or wait for the next night. A temporary answer, such as temporarily_unavailable from Authority or HTTP 503 from the maintained service, keeps Retry.

Other answers the page classifies the same way: unauthorized_client (the client may not use the client-credentials grant), invalid_client (the configured client ID or secret does not match the registration), HTTP 401/403 from the maintained service (its policy did not accept the token) and HTTP 404 (the configured BaseAddress does not serve the maintenance route).

If a database is still growing

1. Check the loop is enabled and the schedule is not paused.
2. Read the service’s own log. The retention pruner explains itself: it names the consumer holding the floor when a stream cannot be reclaimed (consumer '…' holds the floor at seq 0), and says when a stream is bounded by the window alone.
3. A stream held by a consumer that has stopped reporting is logged as a lapsed registration, by name. That consumer is either dead (unregister it) or stuck (fix it) — the growth is attributable either way.

Do not hand-run VACUUM FULL to resolve this. If you believe you need to, that is the defect worth reporting: the bound belongs in product code.