PacksRegistry dual-mode storage cutover and rollback

Sprint: SPRINT_20260722_012 (JOB-12 stages; JOB-13 performs the lab forcing window; JOB-9 later carries the store into JobEngine).

This package is repository-side staging only. No container was recreated, no image was published, and no standing volume was created, mounted, or changed by JOB-12. Execute the procedures below only in an owner-attended deployment or lab window with a reviewed, digest-pinned image.

Staged topology

The current packsregistry-web definition in devops/compose/docker-compose.stella-services.yml selects:

The retained logical volume is packsregistry-upload-data; its explicit physical name is compose_packsregistry-upload-data. Never layer a writable mount over the seed path.

The six explicit default budgets are 1 GiB per pack, 10 GiB and 100,000 objects per tenant, 100 GiB and 1,000,000 objects total, and 512 MiB minimum free space. Operators may lower them through the named PACKSREGISTRY_MAX_* and PACKSREGISTRY_MINIMUM_FREE_BYTES variables after validating that the existing catalog fits. Raising a limit requires the normal capacity review; it is not a recovery shortcut.

devops/compose/docker-compose.jobengine.yml deliberately carries none of these keys or mounts today. jobengine-web is still a health/doctor skeleton and cannot consume the PacksRegistry domain. Advertising ignored storage settings would be a false cutover. The target half is added atomically with JOB-9’s domain and persistence wiring after its owner scope ruling.

Pre-window gates

Abort before recreation unless every item below is true:

1. The candidate image was built from the reviewed commit, is pinned by digest, and contains the dual-store source and migrations proved by JOB-11.
2. Both compose renders below exit zero. The held-back JobEngine overlay stays out of docker-compose.stella-ops.yml, and it contains no PACKSREGISTRY__STORAGE__* setting or packsregistry-upload-data mount.
3. The seed tree exists, contains the intended signed seed-manifest.json, and its recorded tree digest matches the approved generation. Do not repair a mismatch in place.
4. The upload volume resolves to exactly compose_packsregistry-upload-data. If this is the first activation, record the newly created volume identity. If it already exists, record its identity and current byte census; never replace it with an empty volume.
5. The volume has enough free space for the configured total and minimum-free limits, and the service uid can write the upload mount. No write probe may target the seed bind.
6. The database backup/restore point and the old digest-pinned seed-only image are recorded. Forward-only migrations are not rolled back or edited.

Render checks are read-only; they do not start a service. Use the installation’s private env-file so every required variable is present. The committed example file deliberately omits secret values and therefore cannot render this full stack by itself.

docker compose --env-file <operator-private-env-file> \
  -f devops/compose/docker-compose.stella-services.yml config --quiet

docker compose --env-file <operator-private-env-file> \
  -f devops/compose/docker-compose.stella-services.yml \
  -f devops/compose/docker-compose.jobengine.yml config --quiet

First activation

This is a deployment and therefore a confirm-first window, not a repository-side JOB-12 action.

1. Record the seed-tree digest, volume inspection, current image digest, database migration ledger, and current health/API result.
2. Pull or load the reviewed digest-pinned dual-capable image. Do not deploy by a mutable tag.
3. Render the exact effective compose configuration and verify the two roots, the seed :ro bind, the retained upload volume, and all six budgets again.
4. Recreate only packsregistry-web with the approved compose invocation and --force-recreate --no-deps. Do not run down, and never docker compose down -v.
5. Inspect the container mounts. The seed mount must report read-only, the upload mount must report writable, and the upload mount source must resolve to compose_packsregistry-upload-data.
6. Require clean startup reconciliation and migration logs. The existing /healthz endpoint is process health only; it does not earn JOB-12’s richer storage-readiness criterion. Perform the authenticated seed read, upload, restart, origin-aware read, and offline-export forcing calls only in the scheduled JOB-13 lab window.

Abort and enter rollback on any root-overlap/path-escape error, missing or writable seed authority, missing/unwritable upload mount, seed signature or generation mismatch, quarantine, unrecoverable staging conflict, database convergence failure, reservation backlog, capacity watermark, unexpected 400/ 401/403 classification for a storage fault, or any seed-tree digest change.

Rollback without losing uploads

First activation: restore the old seed-only image

Stop new mutations, collect logs/evidence, and recreate only packsregistry-web with the recorded old digest-pinned image and its seed-only configuration. Keep compose_packsregistry-upload-data attached or safely dormant. Uploads created by the dual-capable image may be temporarily unavailable to the old image; they are never deleted, copied, or treated as rollback debris. Do not reverse a migration. Re-check the seed digest and the old image’s seed-read behavior.

Later activation: restore a dual-capable image

Recreate the service with the last known-green dual-capable image and the same seed bind plus the same physical upload volume. Verify reconciliation, catalog origins, artifact digests, and offline export before reopening writes. A new empty volume is not a rollback.

Failure recovery

• 507 capacity full: pause uploads; record logical usage, durable reservations, staging/orphan backlog, volume free bytes, and configured watermarks. Free or expand capacity through an approved storage operation, then let the product’s exact reconciliation recover. Do not delete catalog rows, reservation rows, or blobs by hand and do not weaken tenant/total limits to hide leaked state.
• seed disappearance: stop the service, preserve the upload volume, restore the exact approved signed generation at the same seed path, verify its tree digest, and keep the bind read-only. Never synthesize a replacement manifest.
• quarantine: keep both roots byte-stable and collect the typed conflict, manifest, generation, tenant, pack identity, and digests. Resolve the signed input/catalog conflict through the owning recovery procedure; never overwrite one origin with the other.
• permission or filesystem fault: treat it as a typed storage 503/500, not an authentication or request-shape error. Verify the upload volume ownership, writability, free space, and local-Linux filesystem contract. Do not chmod or remount the seed writable.

JOB-9 same-volume handoff

After JOB-9’s owner ruling and only when jobengine-web actually wires the PacksRegistry domain, its compose change must attach the same seed bind and the same physical compose_packsregistry-upload-data volume at the same container paths. Prove the target mount source by inspection before route swap. The handoff must not copy blob bytes, create a fresh volume, change the physical volume name, or remount the seed writable. Metadata may move to stellaops_jobengine; the blob authority does not.

If target domain wiring and target storage configuration cannot land in one change, abort JOB-9. Do not add ignored PacksRegistry:Storage:ObjectStore:* keys to the skeleton as a placeholder and do not add the held-back overlay to the default stack.

JOB-13 lab prerequisite and outstanding rehearsal

The lab seeder prerequisite is tools/scripts/lab/seed-packsregistry-lab-data.sh. It must run through the product gateway with its existing confirmation and password safeguards; never seed by SQL or migration. Secrets are supplied only through the operator’s named environment variables or mounted secret files and are never printed or copied into evidence.

JOB-12 criterion 2 is NOT RUN by this repository-side package. An isolated scratch stack must still prove, without touching standing containers or volumes: startup/storage readiness, high-water and 507 behavior, seed-only degraded reads, permission-fault classification, restart durability, deterministic offline export/import round-trip, rollback with upload bytes preserved, and a byte-stable seed-tree digest. JOB-13 remains blocked until the program’s stated gates and window scheduling are satisfied.

Re-verify this package

pwsh ./tools/scripts/test-targeted-xunit.ps1 `
  -Project src/__Tests/architecture/StellaOps.Architecture.Contracts.Tests/StellaOps.Architecture.Contracts.Tests.csproj `
  -Class StellaOps.Architecture.Contracts.Tests.PacksRegistryDualModeComposeConformanceTests

pwsh ./tools/scripts/test-targeted-xunit.ps1 `
  -Project src/__Tests/architecture/StellaOps.Architecture.Contracts.Tests/StellaOps.Architecture.Contracts.Tests.csproj `
  -Class StellaOps.Architecture.Contracts.Tests.AppRootWritabilityConformanceTests

Then run tools/scripts/verify-knowledge-base.sh and tools/scripts/validate/check-sprint-status.sh --report. A green render or unit suite is configuration evidence, not proof that the window or scratch rehearsal ran.