JobEngine — building and deploying the staged consolidated stack (JOB-8)

Sprint: SPRINT_20260722_012 (JOB-8 stages; JOB-9 cuts over). Overlay: devops/compose/docker-compose.jobengine.yml — deliberately outside the default config_files list. Publish keys: jobengine-web, jobengine-worker in devops/docker/build-service-publish.sh. Route swap (separate, same window): docs/runbooks/jobengine/jobengine-cutover-job5-route-swap.md.

The three keys this family replaces — scheduler-web, packsregistry-web, packsregistry-worker — keep serving until the JOB-9 window. Nothing here changes the running stack.

What the staged stack serves today (read this before rehearsing)

jobengine-web converges its own database (stellaops_jobengine) and serves:

It does not serve schedules, runs, packs, or the DC-13 submission surface. That is not an oversight and it is not fixable at JOB-8: the S4-extracted application layer (StellaOps.JobEngine.Scheduling) still binds the legacy StellaOps.Scheduler.Persistence, and scheduler-web feeds that DAL the estate’s generic shared-database variable — so referencing the library from the born-new host would give a brand-new service a connection into the shared platform database, which is the exact defect ADR-039 removes. The repoint is JOB-9’s (see the JOB-7 header comment in src/JobEngine/StellaOps.JobEngine.WebService/Program.cs).

Consequence for the window: applying the route swap before that wiring lands produces a uniform 404 farm with green route tests. The swap runbook calls this “the 404 farm”; it is the same prerequisite seen from the deploy side.

Build the images

bash devops/docker/build-service-publish.sh jobengine-web
bash devops/docker/build-service-publish.sh jobengine-worker

Two things about these keys specifically:

Bring up an isolated scratch stack

Never against the live network before the window. Use an isolated project and network:

The overlay no longer stages a PostgreSQL cluster (reshaped 2026-08-14 by SPRINT_20260810_002 PTC-8: the family owns the logical database stellaops_jobengine + the jobengine role on the SHARED installation server, per the ADR-039 P1 topology clarification). A rehearsal therefore supplies its own throwaway server and provisions the database/role with the same script the JOB-9 window uses — which is the point: the rehearsal now exercises the window’s provisioning step instead of a shape the window will not use.

docker network create jobscratch
docker run -d --name jobscratch-pg --network jobscratch \
  -e POSTGRES_USER=stellaops -e POSTGRES_PASSWORD=<scratch-password> \
  <audited digest-pinned postgres image>

PGPASSWORD_SERVICE=<scratch-password> \
  bash tools/scripts/deploy/postgres/provision-service-database.sh \
    stellaops_jobengine jobengine --container jobscratch-pg

INSTANCE_SUFFIX=-jobscratch \
STELLAOPS_JOBENGINE_NETWORK=jobscratch \
STELLAOPS_POSTGRES_JOBENGINE_CONNECTION='Host=jobscratch-pg;Port=5432;Database=stellaops_jobengine;Username=jobengine;Password=<scratch-password>;Maximum Pool Size=16' \
KESTREL_CERT_PASSWORD=<dev-cert-password> \
  docker compose -p jobscratch --env-file /dev/null \
    -f devops/compose/docker-compose.jobengine.yml up -d

STELLAOPS_POSTGRES_JOBENGINE_CONNECTION is required with no fallback — in the compose file and in the service. A JobEngine host that starts against an unintended database is worse than one that does not start. Note the password appears twice (once creating the role, once inside the connection string): compose does not expand ${VAR} inside env-file values (ENV-001), so they are kept in sync by hand, exactly like every other service in the estate. --env-file /dev/null matters too — compose loads devops/compose/.env by default, and that file carries the estate’s live values.

Verifying convergence — a forcing function, not “the container is up”

# 1. The service converged ITS OWN database, from empty. The provisioning script
#    creates the database and role and nothing else (AGENTS.md §2.7): an empty
#    database is the contract. The scratch server is a plain `docker run`
#    container, not a compose service, so exec it directly.
docker exec jobscratch-pg \
  psql -U jobengine -d stellaops_jobengine -c \
  "SELECT schemaname, count(*) FROM pg_tables
   WHERE schemaname IN ('scheduler','scheduler_app','packs','eventing')
   GROUP BY schemaname ORDER BY schemaname"
#    Measured on the first scratch boot (2026-08-06, postgres:18.1), counting
#    with the family baseline's own rule (relkind r/p, non-partition, excluding
#    schema_migrations): scheduler 18, packs 6, eventing 5, and scheduler_app
#    present with 0 tables (it holds only the RLS tenant-context function).
#    18 and 6 are exactly what ConsolidatedJobEngineSchemaTests asserts, so a
#    scratch boot and the unit baseline agree. `pg_tables` as written above also
#    counts schema_migrations, so expect scheduler 19 from that query itself.
#    §2.11 check worth running here: every one of the 24 owned tables is EMPTY on
#    a fresh converge — `0` total rows — and the ledger holds the baseline once.

# 2. The doctor surface applies THIS host's auth posture (the SDK invents none).
#    Unauthenticated must be 401 — a 200 here means the guard was lost.
#
#    NB: the hardened base image ships NEITHER curl NOR wget (verified
#    2026-08-06 — `wget` here fails with "executable file not found in $PATH").
#    Only bash is available, so probe over /dev/tcp, the same fallback
#    devops/docker/healthcheck.sh uses:
docker compose -p jobscratch exec jobengine-web bash -c '
  exec 3<>/dev/tcp/127.0.0.1/8080
  printf "GET /doctor/jobengine-web/checks HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n" >&3
  while IFS= read -r -t 5 line <&3; do case "$line" in HTTP/*|WWW-Authenticate*) echo "$line";; esac; done'

Expected on a correctly-guarded host (measured 2026-08-06):

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer

An authorized /doctor/jobengine-web/checks needs an ops.health token, which needs Authority — i.e. the live stack. On an isolated scratch stack the honest check is the 401 plus the schema census above; the green-checks path is covered by JobEngineDoctorAdoptionTests against a real PostgreSQL.

Both roles must report healthy — including the worker

docker compose -p jobscratch ps must show all three services healthy. The worker is the one to watch, and it is worth knowing why.

The hardened runtime base declares HEALTHCHECK CMD /usr/local/bin/healthcheck.sh, which probes HTTP, and every image built from that base inherits it. A worker that serves no HTTP surface therefore reports unhealthy forever. That is exactly what the first scratch boot measured: jobengine-worker sat unhealthy (“connect: Connection refused”) while every predecessor worker in the running estate — packsregistry-worker included — stayed healthy, because those hosts answer through the shared StellaOps.Worker.Health endpoints.

Fixed at source rather than papered over with a TCP-only probe, which is the ruling the estate already recorded above its own x-worker-health-env anchor (“prefer fixing the app rather than re-adding a TCP-only probe”): StellaOps.JobEngine.Worker now builds a slim web host whose ONLY HTTP surface is /health/liveness + /health/readiness, and the compose key points LIVENESS_PATH / READINESS_PATH at them. Pinned by JobEngineConsolidationConformanceTests.ConsolidatedWorker_AnswersTheHealthcheckItsBaseImageInherits.

If the worker ever goes unhealthy again, check those two env values and that the health endpoints still exist — do not delete the healthcheck.

Redeploy trap (every time, not only the first)

A same-tag image does not swap on up -d. After rebuilding:

docker compose ... up -d --force-recreate --no-deps jobengine-web

Then expect the service to re-send HELLO over Valkey. A gateway 503 "no instances available" immediately after a swap is that handshake, not a broken deploy — one docker restart of the service fixes it. Embedded-SQL or plugin changes need an image rebuild, not a recreate: the migration .sql files are embedded resources and a recreate keeps the old ones.

Doctor registration — leave it OFF until the window

Doctor__Registration__Enabled defaults to false and must stay there until jobengine-web’s Authority client actually holds the platform:doctor:register scope. Enabling it first produces a 403 loop against Platform; the default-off posture exists precisely so an un-granted service degrades to local-only checks (docs/modules/doctor/adoption-guide.md §2). The remaining keys are present as comments in the overlay so the window edits values instead of rediscovering the contract. Two traps live in that guide and both apply here: Authority rejects non-HTTPS token requests (ID2083 — set the explicit TokenEndpoint and mount the CA bundle, which the overlay already does), and the auth client demands an explicit token cache outside Dev/Testing (the host registers the in-memory one).

A third trap is specific to the flip itself. Set the client values, do not just declare them. The reference-adopter wiring guarded Authority/ClientId/ ClientSecret with ?? throw, which accepts an empty string — and compose passes unset variables through as ${VAR:-}. (Both reference adopters were fixed at source on 2026-08-06 by SPRINT_20260722_009 DOC-3, and the shape is now rejected estate-wide by architecture conformance; the operator-facing half below is unchanged.) So flipping Enabled while the client variables resolve to empty would start the host and 403-loop it against Platform with an empty client id, which is exactly what default-off exists to prevent. This host now rejects blank as well as missing values and refuses to start (DoctorRegistration_FailsClosed_OnABlankCredential), so the failure is a loud boot error rather than a quiet loop — but the operator still has to supply real values, and a failed start immediately after the flip means one of the three is empty.

The database owner role — decide this before JOB-9, not during

The baseline enables FORCE ROW LEVEL SECURITY on the tenant-scoped scheduler tables, which binds the table OWNER too. The baseline’s own comment says the owner “must either be granted scheduler_admin or set app.tenant_id per session”. The first half of that does not work, for two independent reasons measured in JobEngineRoleGrantTests:

  1. BYPASSRLS is a role attribute, and attributes are not acquired through role membership — GRANT scheduler_admin TO <owner> leaves isolation fully enforced;
  2. SET ROLE scheduler_admin, which does assume the attribute, then fails on plain schema privileges — the baseline creates that role with no grants at all, so it cannot write the tables it is meant to rescue.

What works is the attribute on the owner role itself (ALTER ROLE <owner> BYPASSRLS, effective on the next connection), or keeping the per-session app.tenant_id posture the application already uses through scheduler_app. Pick one deliberately in the JOB-9 grant step.

Related trap for anyone testing this: a superuser bypasses RLS unconditionally, so any check run as the cluster superuser reports a posture it did not measure.

Rollback

The overlay is not in the default stack, so there is nothing to roll back in the running estate. For a scratch stack:

docker compose -p jobscratch --env-file /dev/null \
  -f devops/compose/docker-compose.jobengine.yml down
docker rm -f jobscratch-pg && docker network rm jobscratch

The scratch server is a plain docker run container rather than a compose service, so down does not reach it — it needs its own line, and removing the container takes its anonymous data with it (it is scratch data by construction).