Runbook — routine database maintenance

Audience: operators of a Stella Ops installation. Short version: there is nothing for you to do. Routine database maintenance — retention pruning and planner-statistics refresh — runs nightly through the product’s own scheduler on every installation. This page exists so you can see what it does, confirm it is running, and change it if you want to.

Owner ruling, 2026-08-10: Stella Ops must be self-serviceable. No installation may depend on an operator (or on us) running psql rituals or authorizing routine retention. If you find yourself hand-running SQL to keep a database from growing, that is a defect — please report it.

What runs, and when

01:00 is deliberately before the 02:00 nightly scan: reclaim first, then scan against a database that is not carrying the backlog.

Each service’s pass does two things, both bounded:

1. Retention pruning — reclaims event envelopes that every consumer has durably processed AND that are older than the service’s declared window. Nothing unconsumed is ever deleted; see the eventing contract.
2. ANALYZEof the hot tables, so the query planner’s statistics describe the table as it now is.

It never takes an exclusive lock. VACUUM FULL and CLUSTER take an ACCESS EXCLUSIVE lock on tables every consumer touches on every event; a nightly job must never be able to freeze the estate on its own initiative. If you ask for one (vacuum: true), the service answers vacuumPerformed: false and says why, rather than silently doing less than you asked.

Is it running?

The trigger loop says which mode it is in at startup, so “nothing fires” is never a mystery:

docker logs stellaops-scheduler-web 2>&1 | grep -i "cron triggering"

• Cron triggering enabled: polling every … — running.
• Cron triggering is DISABLED (Scheduler:Worker:Trigger:Enabled=false); stored schedules will NOT fire. — not running. Runs can still be created by hand through the API, but nothing is scheduled.

Then look at the runs themselves. A cron run is an ordinary run: it appears in GET /api/v1/scheduler/runs, it can be cancelled with POST /api/v1/scheduler/runs/{runId}/cancel, and it produces an audit event — exactly like a run you started yourself.

Turning it on

As of this release the loop ships DISABLED by default. That is a staging posture for the first supervised activation, not the intended end state: the ruling is nightly-by-default on every installation, and a later release flips it. Enabling it wakes every schedule that already exists on your estate, so do it deliberately and watch the first night.

# devops/compose/.env
SCHEDULER_TRIGGER_ENABLED=true

Then recreate the scheduler:

docker compose … up -d --force-recreate --no-deps scheduler-web

Changing it

Everything below is ordinary configuration; none of it needs a rebuild.

To change when maintenance runs, edit the schedule itself — it is a normal schedule:

PATCH /api/v1/scheduler/schedules/sys-default-nightly-db-maintenance
{ "cronExpression": "0 3 * * *" }

Your edit survives restarts. The system schedules are created once if absent and never overwritten.

There is no connection string here, and that is deliberate. The scheduler never opens a connection to any service’s database; it asks each service to maintain its own (CoC §8.2). A service that has not implemented the maintenance contract simply is not listed.

The misfire grace, in plain terms

If the scheduler is down when a job was due, should it run late or wait for the next slot? MisfireGrace is that answer, and it bounds catch-up to at most one run per schedule — never a backfill of everything missed.

• Down for 5 minutes across a 01:00 job, grace 10 minutes ⇒ it runs at start-up.
• Down for an hour ⇒ 01:00 is skipped and it runs tomorrow at 01:00.
• Down for a week ⇒ still just tomorrow at 01:00. It never replays the week.

Reading the result

The job logs one line per adopter, with numbers the owning service measured against its own database:

Maintenance completed for 'vulnerabilities': 50000 row(s) reclaimed, 41943040 byte(s) released.

Two things that look like problems and are not:

• batchLimitReached: true. Each pass is bounded so it cannot flood the WAL. A large first backlog drains over several nights. Watch the age of the oldest row rising, not only the row count falling.
• Bytes released is zero or negative. Deleting rows leaves dead tuples that only autovacuum returns to the free space map, and ingest keeps writing throughout. A pass that reclaimed 50,000 rows while the table grew is a real and honest signal. We report the raw before/after rather than clamping it to a flattering number.

Running it now, by hand

Not required — but if you want a pass immediately, trigger the schedule through the ordinary run API rather than touching the database:

POST /api/v1/scheduler/runs
{ "scheduleId": "sys-default-nightly-db-maintenance", "reason": { "trigger": "manual" } }

Or call one service’s maintenance directly (needs ops.health):

curl -sX POST -H "Authorization: Bearer $TOKEN" \
  https://stella-ops.local/api/vulnerabilities/v1/maintenance/runs \
  -H 'content-type: application/json' -d '{"reason":"manual"}'

If a database is still growing

1. Check the loop is enabled and the schedule is not paused.
2. Read the service’s own log. The retention pruner explains itself: it names the consumer holding the floor when a stream cannot be reclaimed (consumer '…' holds the floor at seq 0), and says when a stream is bounded by the window alone.
3. A stream held by a consumer that has stopped reporting is logged as a lapsed registration, by name. That consumer is either dead (unregister it) or stuck (fix it) — the growth is attributable either way.

Do not hand-run VACUUM FULL to resolve this. If you believe you need to, that is the defect worth reporting: the bound belongs in product code.