OfflineKit — building and deploying the staged consolidated stack (OK-9)

Sprint: SPRINT_20260722_025 (OK-8 adopts the doctor surface and registers the keys; OK-9 stages this stack; OK-10 cuts over and deletes).

HISTORICAL — the window this stages for has run. OK-10 cut over on 2026-09-13: export-web, export-worker and airgap-controller are stopped (docker ps -a: Exited(0)), the four legacy publish keys are gone from devops/docker/build-service-publish.sh, and docker-compose.offlinekit-consolidated.yml is in the live compose chain rather than outside it. Read the “Nothing here changes the running stack” line below as a statement about the OK-9 staging step, not about the estate today. The document is left otherwise intact: it is the build-and-stage procedure, and the window evidence in docs/implplan/_evidence/20260913-ok10-consolidation-window/ reads against it.

Nothing here changes the running stack. export-web, export-worker and airgap-controller keep serving from docker-compose.stella-services.yml (lines 3519, 3712 and 3033) until the OK-10 window; airgap-time (line 3095) stays behind profiles: ["airgap"].

Read this first: the delta is a MIGRATION of three live services, not a birth

Design doc §10 said “the flagship stack does not deploy this family at all”. The observation under it is true — docker-compose.stella-ops.yml contains no service block for these hosts — and the inference is false, because that file is include-based (:30-32 includes docker-compose.stella-services.yml). Corrected at OK-6; repeated here because it changes what this runbook is for. Three of the four legacy services start by default, so:

• the OK-9 window stops live services, it does not add new ones beside nothing;
• the OK-6 route swap is a live-traffic change;
• only the two OfflineKit hosts are genuinely absent today.

Why this family DOES register publish keys at S8 — and the one flip that must wait for OK-10

Findings, JobEngine and the vulnerability hub registered consolidated keys at their S8 because their hosts were born new (zero violation pairs, so the keys are free). AdvisoryAI could not, because a second key over one continuation host re-counts the same source graph, and Graph needed neither because it chose continuation. This family is in the born-new case and needs the keys more than any sibling did, because its remaining blocker is LIVE: OK-4’s boundary criterion is still open on the export-*|policy pair (D-OK4-9, owned by SPRINT_20260722_007), so what must stop someone composing the export domain into these hosts is the gate, not a review. Before OK-8 the boundary report did not walk these hosts at all — the report only walks keys the publish helper declares — and the sprint recorded twice that “offlinekit-web has no publish key so the boundary checker would not even see it”.

Measured on registration: offlinekit-web 38 projects in closure, offlinekit-worker 3, both with ZERO violation pairs; estate 55 keys / 135 pairs / 0 unpinned / 0 stale / 0 grown.

One flip that looks like OK-9’s job is not, and the manifest note said the wrong thing until OK-8 corrected it. The two libraries OK-4 lifted into this family’s directory — src/OfflineKit/__Libraries/StellaOps.ExportCenter.Application and .WorkerRuntime — are owned by export-center, and the note used to say they re-classify to offlinekit “with OK-9”. They re-classify at OK-10. Measured 2026-08-06 by flipping both owners on the working tree, running the gate, and restoring:

violation pairs : 137   (from 135)
unpinned pairs  : 2
  UNPINNED violation: export-web -> offlinekit (1 projects;
    witness: …/StellaOps.ExportCenter.WebService.csproj
          -> src/OfflineKit/__Libraries/StellaOps.ExportCenter.Application/…csproj)
  UNPINNED violation: export-worker -> offlinekit (2 projects;
    witness: …/StellaOps.ExportCenter.Worker.csproj
          -> src/OfflineKit/__Libraries/StellaOps.ExportCenter.WorkerRuntime/…csproj)
exit 1

The cause is structural, not tuning: export-web/export-worker are still deployables and still compile those libraries, so any owner other than export-center makes their own hosts’ edges cross-family. Restored state re-verified: 135 pairs, 0 unpinned, OK. The re-classification is correct only once OK-10 deletes the legacy hosts, their compose keys and their publish keys. TheLiftedExportLibraries_StayOwnedByExportCenterUntilTheLegacyKeysDie pins it — and pins it conditionally, keyed on whether the legacy publish keys still exist, so it flips its own expectation at OK-10 instead of needing an edit.

What the staged stack serves today (read this before rehearsing)

offlinekit-web now serves these source routes:

• GET /health/ready — anonymous (the container healthcheck and the gateway poll it);
• GET /doctor/offlinekit/checks — doctor-check/v1, ten checks, gated on ops.health;
• /api/offlinekit/v1/{import,status,manifest,validate} — the canonical verified-carrier plane, including the bounded Scanner artifact-BOM activation slice.

offlinekit-worker serves nothing. It converges the family’s database and idles.

The export run engine and distribution surfaces are not composed into either host, because OK-4’s boundary criterion is still blocked on export-*|policy — re-measured at OK-8 against HEAD and still live (ExportCenter.Application → StellaOps.Policy.Exceptions, ExportCenter.Core → StellaOps.Policy). Swapping the OK-6 gateway group against this service today still takes the export/distribution surface down; it no longer lacks the canonical import gate.

The window’s ten checks, and what each one is for

Run GET /doctor/offlinekit/checks with an ops.health identity. Every check measures a precondition of this window that the sprint file otherwise carries only as prose, and each measures a silent failure.

The evidence never contains the signing key. It reports the SOURCE and the fact of configuration only; a doctor result must be safe to ship in a support bundle.

The two gateway preconditions, which are the window’s real risks

1. Identity (F-OK6-2) — and it INVERTS every sibling’s finding. All five siblings concluded PreserveAuthHeaders was inert because their replaced routes were regex. In the deployed config this family’s four export-center routes are non-regex with the flag set, so they enter JwtPassthroughPrefixes and the Bearer is forwarded to export-web today; the deployed ApprovedAuthPassthroughPrefixes even carries a ninth entry (/export-center) the service default lacks. The OK-6 group is a regex route, which cannot enter that list — so the swap changes the identity contract instead of preserving it, and identity then arrives only in the gateway-signed envelope.

OK-8 wired UseIdentityEnvelopeAuthentication() on offlinekit-web. Measured while doing it: neither src/ExportCenter nor src/OfflineKit contained a single call to it, so the predecessor host has never had this path at all. The middleware does not fail closed — with no signing key it logs one warning and lets the request continue unauthenticated (IdentityEnvelopeMiddlewareExtensions.cs:71-75), which makes a missing key a total 401 outage whose only symptom is a per-request warning line. The overlay therefore takes STELLAOPS_IDENTITY_ENVELOPE_SIGNING_KEY with :?, not :-, and doctor.offlinekit.gateway.envelope-identity reports it.

2. Router registration — a precondition no sibling faced at all. A gateway Microservice route resolves its target through the service registry by HELLO ServiceName, not by DNS. export-web registers as exportcenter; StellaOps.OfflineKit.WebService called AddRouterMicroservice nowhere, so it advertised nothing, and pointing ^/api/offlinekit/v1(.*) at an unregistered name answers 503 target-microservice-unavailable for the whole plane. Every sibling’s swap targeted a live, already-registered host.

OK-8 wired the call. It is config-gated: with no enabled Router section it returns false and registers nothing, so wiring changed no behaviour. Being wired is therefore not the same as being registered — doctor.offlinekit.gateway.router-registration reports the boolean the helper actually returned, and it is red until an operator enables the Router section in the window. The compose alias existing does not help; the alias is not what the route resolves by.

Build the images

First-ever builds for both keys.

CORRECTED 2026-08-17 (OK-9 rehearsal, D-OK9-4). This section used to say the builds “layer over the identical-runtime predecessor bases (stellaops/export-web:dev and stellaops/export-worker:dev)”. That is wrong twice, and believing it is actively dangerous:

• Since SPRINT_20260727_002 PKG-1 the default STELLAOPS_PUBLISH_BASE_MODE=clean builds every service FROM a service-INDEPENDENT base flavour (stellaops/service-base:<flavour>-<rev>). The third field of the key triple in build-service-publish.sh is consulted only by the legacy escape hatch. Measured: both builds resolved stellaops/service-base:default-e5c197de300a.
• stellaops/export-web:dev does not exist on the reference host (the live image is tagged bl2-9796c887-20260724). An executor who believed the old text would hit a “missing base” and might switch to legacy — which is exactly the pre-PKG-1 behaviour that produced unbounded layer growth (measured at up to 91.7% dead bytes) and which the script’s own header says never to release from. Do not set STELLAOPS_PUBLISH_BASE_MODE=legacy to work around a missing :dev tag.

Measured result of the first-ever builds: offlinekit-web 10 layers / 273 MB, offlinekit-worker 10 layers / 258 MB, both inside check-image-budget.sh’s 20-layer / 1000 MB limits.

devops/docker/build-service-publish.sh offlinekit-web
devops/docker/build-service-publish.sh offlinekit-worker

Rehearse in isolation (safe at any time)

The overlay defaults its network to the live stellaops name, so always set a scratch network for a rehearsal or the containers join the live stack.

CORRECTED 2026-08-17 (D-OK9-5): use a throwaway postgres on the scratch network, not a scratch database on the shared server. The previous recipe asked for both a scratch NETWORK and a scratch DATABASE on the live stellaops-postgres, which cannot both hold: a container on a fresh bridge network cannot reach the live postgres by container IP, and its published port is bound to the loopback alias 127.1.1.1:5432, which host.docker.internal does not reach either. The only literal readings are “attach the rehearsal to the live network” (forbidden two lines up) or “mutate the live postgres container’s networking”. A throwaway postgres is safer AND a better proof — nothing is borrowed from the estate, so “converges on a fresh database” means exactly that.

Compose must OWN the network (it refuses a pre-existing one for want of its label), so create the network via compose first, then start postgres on it, then start the services:

cd devops/compose
export STELLAOPS_OFFLINEKIT_NETWORK=okscratch
export STELLAOPS_POSTGRES_OFFLINEKIT_CONNECTION='Host=okscratch-postgres;Port=5432;Database=stellaops_offlinekit;Username=stellaops;Password=stellaops'
export STELLAOPS_IDENTITY_ENVELOPE_SIGNING_KEY='...'
export KESTREL_CERT_PASSWORD='...'

# 1. compose creates the labelled network + both containers (started at step 3)
docker compose -p okscratch -f docker-compose.offlinekit-consolidated.yml create

# 2. the throwaway database, on that network
docker run -d --name okscratch-postgres --network okscratch \
  -e POSTGRES_USER=stellaops -e POSTGRES_PASSWORD=stellaops \
  -e POSTGRES_DB=stellaops_offlinekit postgres:18.1

# 3. now the hosts can converge their schema on first boot
docker compose -p okscratch -f docker-compose.offlinekit-consolidated.yml start

Teardown (leave nothing behind):

docker compose -p okscratch -f docker-compose.offlinekit-consolidated.yml down
docker rm -f okscratch-postgres && docker network rm okscratch

Expect: both containers up, offlinekit-web healthy, and — with an ops.health identity — /doctor/offlinekit/checks reporting the two gateway checks red (no Router section, and the envelope key present only if you set it). That red set is correct pre-window; a green router-registration before the window would mean the host is advertising into the live registry from a rehearsal.

MEASURED 2026-08-17, and one expectation above is FALSE. Actual results:

D-OK9-3, found here and since FIXED. On the first pass offlinekit-worker reported unhealthy forever while its process ran correctly (“Application started” in its own log): the inherited healthcheck probes /dev/tcp/127.0.0.1/8080 and the host was a Microsoft.NET.Sdk.Worker serving no HTTP surface. It was the class’s ONLY such host — both ExportCenter.Worker and JobEngine.Worker are Microsoft.NET.Sdk.Web — and JobEngine had already answered the identical defect at JOB-8 with the shared StellaOps.Worker.Health endpoints. That pattern is now adopted: the worker hosts /health/liveness and /health/readiness and nothing else (verified: /doctor/offlinekit/checks, /v1/exports and /api/offlinekit/v1/exports all 404). Re-running this rehearsal after the fix shows both containers healthy.

The ops.health step is NOT performable in an isolated rehearsal: minting that identity needs an Authority the scratch network deliberately cannot reach. Run it in the window, or attach a token out of band.

Both hosts fail closed without STELLAOPS_POSTGRES_OFFLINEKIT_CONNECTION — there is no generic shared-connection fallback, which is the X5-class defect the merge fixes (both legacy hosts ride the generic ConnectionStrings__Default today).

What compose cannot stage, and what owns each piece

Window order (OK-9 → OK-10)

1. Create stellaops_offlinekit and its owner role; set the connection variable.
2. Bring the two hosts up on the live network; confirm /health/ready and /doctor/offlinekit/checks.
3. Enable the Router section so offlinekit-web registers; confirm doctor.offlinekit.gateway.router-registration goes green. Do not proceed while it is red.
4. Confirm doctor.offlinekit.gateway.envelope-identity is green.
5. Only then apply the OK-6 route swap (docs/runbooks/airgap/offlinekit-cutover-ok6-route-swap.md) — one routing change, DC-19.
6. Stop export-web, export-worker, airgap-controller; copy export_center with parity; flip the SCR-3 tenant replication in the same step.
7. Re-read /doctor/offlinekit/checks: tenancy.resolution-source must be Info (local replica), and export-engine.inventory must report the copied row counts.