JobEngine retention and custody
JobEngine separates completed scheduling/delivery mechanics from durable approval, dispatch, pack and cryptographic evidence. Forward migrations 010–012 retire unused empty declarations, implement terminal cleanup and record each table’s class. The P13 source inventory and operational proof registry bind each operational table to its real predicate, window, budget, Doctor check and active-row safety test.
Keeper configuration and scope
The consolidated web host registers JobEngineRetentionService after startup migration registration. Settings are under JobEngine:Retention:
| Setting | Default | Meaning |
|---|---|---|
Enabled | true | Run scheduled cleanup. |
TerminalRetentionDays | 90 | Age window for completed mechanics. |
FailureSignatureRetentionDays | 90 | Existing resolved-signature retention intent, now enforced by the scoped keeper. |
JobHistoryRetentionDays | 365 | Archived job outcome window; cannot be shorter than terminal retention. |
MaximumTerminalRowsPerScope | 10000 | Maximum eligible terminal rows per tenant, or globally for worker registrations. |
BatchSize | 1000 | Maximum deleted rows per routine call. |
MaxBatchesPerSweep | 10 | Finite call budget per table and scope. |
SweepInterval | 1.00:00:00 | Daily cadence. |
StartDelay | 00:05:00 | Delay before the first sweep. |
Positive windows, budgets and interval are required; the start delay may be zero. Each sweep iterates the existing Scheduler:Worker:Trigger:Tenants scope (whose documented unset default is default) and tenant identifiers present in owned pack metadata, storage events or outbox rows. Retained pack events keep those tenants discoverable after their outboxes drain. This does not claim discovery of arbitrary scheduler-only tenants outside the instance’s configured worker scope.
Every scheduler payload operation opens a tenant-scoped connection and the SQL routine verifies that its tenant argument matches the session. No runtime RLS bypass, new tenant directory or privilege grant is introduced. One tenant failure is logged and other scopes continue. The worker registry is global and has its own cleanup.
What can expire
| Table | Terminal condition and safety boundary |
|---|---|
scheduler.jobs | Succeeded, failed, canceled or timed-out, with a completion time. Keep live leases and trigger-referenced jobs. Deletion and insertion of the outcome into job_history are one statement. |
scheduler.runs | Completed, error or cancelled with finished_at. Planning, queued and running work survives. |
scheduler.graph_jobs | Completed, failed or canceled; age is measured from the last update. |
scheduler.policy_run_jobs | Completed, failed or cancelled, with no live lease. Age is measured from the last update, covering cancellation paths that do not set completed_at. |
scheduler.estate_run_queue | Acknowledged with both completion time and outcome, and no delivery lease. Durable dispatch/approval authority remains. |
packs.pack_storage_outbox | Published acknowledgement exists. Every unpublished envelope remains; this work does not implement the separate publisher. |
scheduler.failure_signatures | Resolved with resolved_at. Unresolved, investigating and wont-fix records remain. |
scheduler.workers | Stopped and idle, measured from the last heartbeat. Active, draining and busy registrations remain. |
Age or excess eligible row count can select a terminal row. Deletion rechecks its current state, timestamp and applicable lease/reference guards so a refreshed or reopened row is not deleted from an old candidate snapshot. SQL state aliases express timestamp-based acknowledgements without adding a second persisted state column.
Schedule summaries reconstruct only the latest retained run of each non-retired schedule. Retirement excludes that schedule from summary reads without deleting its runs. Run expiry can select an older retained run; deleting the last retained run removes its summary and lastRunAt. These fields are not lifetime execution ledgers, and null does not prove that a schedule has never run. Delayed projection deliveries cannot recreate deleted history because they never write summary state. The durable summary service keeps no cache or separate fence table. See the summary lifecycle contract.
job_history has its own completion-time window. Pruning jobs ends the live queue’s deduplication lifetime for those rows; durable authorization and claim-once records have separate custody and do not expire with transport state.
Records that remain
The keeper does not delete estate approvals, allowed/refused decision audits, claim-once dispatch authority, rate-limit dispatch occurrences, authored schedules or triggers, captured impact results, exception policy, HLC logs/heads/anchors, pack artifacts, signed attestations, signature floors/history, seed validation evidence or artifact-readiness facts.
Physical upload reservations remain authoritative capacity claims. Only the existing fenced finalization/stale-operation recovery can release an exact reservation; age alone cannot reclaim bytes from an active writer. Mirror parity rows retain the captured comparison and notes rather than claiming a later remote comparison reconstructs the original observation.
Diagnostics
doctor.jobengine.retention.terminal-budgets runs through /doctor/jobengine-web/checks under the host’s existing ops.health policy. It reports the scope count, configured windows and budgets, maximum eligible terminal rows per scope, expired rows, protected rows and incomplete terminal metadata for each of the eight operational tables. A disabled keeper, expired eligible rows, exceeded row budget or incomplete terminal metadata produces an unhealthy result. Protected active work remains intact. No tenant identifiers or payloads are included in the evidence.
Empty-table retirement
doctor_trends, execution_logs, graph_job_events and run_summaries have no current production readers/writers. Migration 010 refuses any populated target. It locks all targets and checks them as the table owner in one atomic statement, temporarily removing FORCE RLS only inside that locked migration. This prevents a tenant filter from hiding another tenant’s data. Any refusal rolls back the RLS flags as well as the attempted change; successful empty retirement drops the tables without CASCADE. Runtime policies do not relax.
Verification and rollout
Verified against source commit 923eb4cf543ea13402d2146d44dd57dfc7ee0e72. The verification receipt records the executed checks, negative controls and signed producer staging.
JobEngineRetentionTests exercises the real SQL/store/keeper under a non-superuser, NOBYPASSRLS owner, including active work, tenant isolation, finite budgets, queue history, leases, preserved source records, concurrent timestamp refresh, and hidden-row retirement refusal. JobEngineDoctorAdoptionTests verifies the real host’s 401/403/200 boundary and Doctor registration. Run the consolidated schema and adjacent queue tests with tools/scripts/test-targeted-xunit.ps1.
The Program sprint records the verified source and exact results. Source/integration acceptance is separate from rollout. Rebuild the web/worker images and repackage the estate-doctor producer bundle using the staging procedure when promoting this persistence assembly; the producer bundle must still exclude host-shared assemblies. No live image or mounted bundle is replaced by RFC-7 validation.
Owner capacity monitoring (DIP-17)
The consolidated PostgreSQL web host adds a leased probe on its explicit owner pool. Its physical database budget is stellaops_jobengine; its established migration schema is scheduler. Migration 015_capacity_probe_receipt.sql creates scheduler.capacity_probe_receipt, and every insert retains the newest 90 receipts. doctor.jobengine.capacity.budget measures the ten registered entries in scheduler and packs, rejecting absent required measurements, missing/stale receipts and stop-line breaches. The explicit local harness without PostgreSQL does not register a database probe.
The capacity guide records the initial envelope and its limitations. Monitoring does not complete all 28 identity declarations or intercept writers. Its pause flag means pause required; no queue, approval, pack, reservation or evidence disposal rule changes. JobEngineDoctorAdoptionTests exercises the real host’s automatic migration and capacity check; run it through the targeted wrapper with -BuildProjectReferences.