Image Rebuild Order Runbook

Audience

Operators and autonomous agents who need to rebuild one or more Stella Ops service images in a running compose stack. Specifically: anyone who has just changed code in a shared library (auth, router, plugin host, persistence) and wants the running stack to pick up the change without manual debugging of “why does the new image still behave like the old one.”

This runbook collects three independent gotchas that multiple 2026-05-29 sprints (046, 049, 050, 055, 060, 066) rediscovered ad-hoc. The cost of each rediscovery is roughly 30-90 minutes of agent or operator time; documenting them here is the cheapest possible mitigation.

Scope (verified 2026-09-26 against 942a18727c). This runbook is for the source-built development and lab stack under devops/compose/. A customer install of the release bundle never rebuilds images: it runs the digest-pinned images named in release-manifest.yaml, and docker compose starts them in the order the bundle’s depends_on declares. Release images are built by devops/release/bundle/tools/build-and-publish.sh, which calls devops/docker/build-service-publish.sh for every stellaops/* image the bundle’s docker-compose.yml references. See the release bundle section below.

Prerequisites

When to use this runbook

Operator situationAction
Single-service code change (no shared lib touched, no auth touched)Targeted rebuild of just that service + up -d --force-recreate <service>
Shared lib change (e.g. src/__Libraries/StellaOps.Auth.*, src/Router/__Libraries/)Full ordered rebuild per the order in the next section
Authority schema or signing-key changeAuthority FIRST, then all consumers (because consumers cache the JWKS validation keys in-process at startup)
Release Orchestrator execution bundle-only changeRun devops/build/package-execution-plugins.ps1 for the affected profile and remount/recreate agent-core; do not rebuild agent-core unless host runtime code, Dockerfile tools, or shared Plugin Host code changed
Plugin host source change (src/Plugin/StellaOps.Plugin.Host/)Plugin host + all plugin-consuming services (release-orchestrator, agent-core, scanner-worker) — together so the loader contract matches across host and consumers
Compose file or env-file change (no image change)docker compose up -d is sufficient — but verify with docker inspect because compose has been observed silently no-op’ing layered FS swaps (see Caveat 1 below)
Angular/Console change — including any API path prefix or base URL (src/Web/StellaOps.Web/src/app/core/api/api-path-prefixes.ts)No image rebuild will help. The Console is a pre-built static bundle bind-mounted by absolute path, so it is invisible to every rebuild in this runbook — see Caveat 5

Quick reference: the canonical rebuild order

When a change touches auth, router, or any shared library that crosses service boundaries, the canonical rebuild + restart order is:

  1. authority— JWT issuer / OIDC. Must be ready before any consumer restarts, otherwise consumers cache the OLD JWKS at boot.
  2. scanner-worker— registers per-tenant token handlers (Sprint 048 / Sprint 051). Must boot AFTER authority so the per-tenant client credentials flow against a current issuer.
  3. integrations-web— owns the registry credential resolver chain (Sprint 027 / Sprint 046). Must boot AFTER scanner-worker so the scanner-worker’s IntegrationAwareRegistryCredentialProvider (per-process cache) is keyed against the same integration-row state.
  4. Other consumer services (router-gateway, release-orchestrator, and the rest) in any order, but AFTER the three above. findings-web and the AdvisoryAI hosts are not in the base chain; they run from their own overlays (docker-compose.findings.yml, docker-compose.advisoryai.yml) and are restarted through those files.

This ordering exists because Stella’s bearer-token validation, integration catalog, and per-process credential caches all warm up at container startup; rebooting a downstream consumer before its upstream is ready bakes a stale view into the in-process state until the next restart.

Procedure

Step 1 — Select and verify the affected service window

Inspect the source difference and owning service’s runtime before selecting a rebuild. Build from an isolated committed checkout. Read each running container’s recorded Compose chain and render only the reviewed service candidate. Preserve its environment, mounts, physical volume identities and outgoing image. See estate posture reconciliation.

An Authority window completes readiness and token probes before affected consumers restart. A shared-library change does not authorize stopping the whole estate or removing unrelated containers. Recreate each selected service with --no-deps.

Step 2 — Build committed service source

Use the targeted publish-then-COPY builder in the isolated checkout:

IMAGE_TAG=reviewed-window bash devops/docker/build-service-publish.sh authority

Select the resulting image by digest in the owning durable profile. Verify its buildinfo source and clean checkout state. The builder publishes transitive source before constructing the image; verify the produced payload rather than assuming that disabling Docker’s cache repairs a source or packaging problem.

For agent-core specifically (Sprint 066 caveat), use:

& 'C:\Program Files\Git\bin\bash.exe' -c 'bash devops/docker/build-service-publish.sh agent-core'

— and confirm the new digest with docker images stellaops/agent-core --no-trunc.

Execution plugin bundle updates are separate from agent-core image rebuilds. For local/offline-kit evidence, build signed bundle profiles first:

powershell -NoProfile -ExecutionPolicy Bypass -File .\devops\build\package-execution-plugins.ps1 -Profile default
powershell -NoProfile -ExecutionPolicy Bypass -File .\devops\build\package-execution-plugins.ps1 -Profile optional-build-script

Then render the profile-gated agent overlay through the compose wrapper:

$env:COMPOSE_PROFILES = 'agents'
$env:COMPOSE_EXTRA_FILES = 'docker-compose.plugins.execution.yml'
.\devops\compose\scripts\compose-cli.ps1 config --format json

The agent-core image should contain the host runtime, /app/tools/StellaOps.Plugin.Host, and required CLIs only. Execution payload files such as plugin.json, plugin.signature.json, StellaOps.Agent.BuildDocker.dll, StellaOps.Agent.BuildScript.dll, StellaOps.Agent.DeployCompose.dll, and StellaOps.Agent.DeployNative.dll must come from read-only mounted bundle roots. When a local image exists, verify that with:

powershell -NoProfile -ExecutionPolicy Bypass -File .\devops\build\audit-pluginized-image.ps1 -Image stellaops/agent-core:dev -Service agent-core -FailOnPluginRootPayload

Step 3 — Restart in the canonical order with --force-recreate

Use the reviewed ordered -f chain, project name and project directory for that service, with its candidate digest selected by the final owning image profile:

docker compose -p stellaops --project-directory devops/compose \
  -f <reviewed-first-file> -f <reviewed-last-file> \
  up -d --force-recreate --no-deps --pull never <service-key>

Name one selected service or one reviewed family window. An unscoped --force-recreate affects unrelated services and is not an idempotent no-op.

--force-recreate is REQUIRED because container restart alone does NOT swap the layered filesystem to a freshly-tagged image — docker compose restart only stops and starts the existing container against its pinned image digest. The newly-baked image is then inert until the container is destroyed and re-created. This was rediscovered by the Sprint 066 S066-001 agent (commit ef5120dd4c) when the rebuilt agent-core image with docker-buildx baked in appeared not to ship the new binary until --force-recreate ran.

Step 4 — Verify each service picked up the new image

docker inspect stellaops-authority --format '{{.Image}}'
docker inspect stellaops-scanner-worker --format '{{.Image}}'
docker inspect stellaops-integrations-web --format '{{.Image}}'

Cross-check the returned image SHA against docker images stellaops/authority --no-trunc --format '{{.ID}}'. If they match the newly-baked digest, the swap landed. If not, repeat Step 3 with the explicit service name AND --force-recreate.

For the agent-core image specifically, confirm the bundled AGENT_CORE_CAPABILITIES env-file is a list of 10 capability strings (Sprint 066 caveat):

docker exec stellaops-agent-core printenv AGENT_CORE_CAPABILITIES

Empty or short lists indicate the env-file was not picked up at boot and the agent will fail to register its capability set with the orchestrator.

Step 5 — Smoke check

Run the gateway sentinel curl per tenant-onboarding.md (or a simpler curl -k https://stella-ops.local/.well-known/openid-configuration against authority) to confirm bearer-token issuance still works against the rebuilt authority.

Caveats (the costly rediscoveries)

Caveat 1 — docker compose restart does NOT pick up new images

Rediscovered by Sprint 066 S066-001 (commit ef5120dd4c) — the rebuilt agent-core image with docker-buildx baked in appeared inert until --force-recreate ran. A restart preserves both image and container environment. Environment changes also require recreation. Verify the selected digest and actual container image after the named service swap.

Caveat 2 — AGENT_CORE_CAPABILITIES env-file 10-capability requirement

Rediscovered by Sprint 050 (commit referenced in SPRINT_20260529_050) — docker-compose.lab-plugin-bundles.yml set AGENT_CORE_CAPABILITIES to a bare deploy.compose string. The compose plugin’s manifest declares deploy.compose.{pull,preflight,up,...} as 10 separate capabilities; the orchestrator’s catalog resolves by alias namespace, so the bare string fails silently — the plugin is discovered, signature-verified, and trust-mapped, but never instantiated because no capability matched. Always pass the full capability list (10 entries for the standard compose deploy plugin) in AGENT_CORE_CAPABILITIES.

Caveat 3 — In-process auth state caches stale token-validation keys

Rediscovered by Sprint 060 S060-001 (PluginTrustRoot regression discovery, commit 5df04f6b02 was the trigger). Authority issues JWTs signed with a per-installation key. Each consumer service fetches the JWKS at startup and caches it in-process. If you rebuild + restart authority, then rebuild a consumer that boots BEFORE authority is fully healthy, the consumer caches the OLD JWKS (or fails to bootstrap if authority is fully down). Symptoms: 401 Unauthorized on previously-working endpoints with a signature failed reason in the audit log. Resolution: stop the consumer, wait for authority /health to return 200, restart the consumer.

Caveat 4 — verify the transitive compiled payload

The targeted builder compiles the service’s transitive project graph before COPY. Check the clean source stamp, required assemblies and packaged plugin contracts. A missing or stale publish input is a build/packaging defect; disabling Docker’s cache alone does not prove that the correct transitive assembly was published.

Caveat 5 — the Console is a bind-mounted bundle, so no image rebuild reaches it

Rediscovered by SPRINT_20260722_025 OK-10 (the OfflineKit consolidation window, 2026-09-13). The gateway route group moved the export plane from the bare /v1/exports to /api/offlinekit/v1/exports, and api-path-prefixes.ts was updated in the same change — but the deployed Console kept calling the old path for a further day, because the running Console is not an image this stack rebuilds. router-gateway serves /app/wwwroot from an absolute host path pinned in devops/compose/bin8/gateway-forward.yml, a git-ignored local file created from the committed gateway-forward.yml.example:

devops/runtime/console/<sha>/browser  ->  /app/wwwroot

Nothing in build-all.sh, --no-cache, or up -d --force-recreate touches that directory. The symptom is the worst kind: the API is correct, the gateway is correct, the source is correct, and the browser still 404s on a path no one can find in the tree — because the string lives in a minified chunk-*.js under devops/runtime/console/, not in src/.

The procedure.

  1. Build from src/Web/StellaOps.Web: npx ng build.
  2. Publish the output to a NEW devops/runtime/console/<sha>/browser — a new directory, never an in-place overwrite, so the previous bundle stays as the rollback.
  3. Move the pin in devops/compose/bin8/gateway-forward.yml to the new path.
  4. Roll it out with devops/compose/scripts/redeploy-frontdoor-ui.ps1 — the only executable recipe for this. It also reapplies the Authority seed migrations and reconciles the Platform envsettings Scope row, which is why it is preferred over a bare up -d. Use -SkipBuild when only the console bundle changed.

Verify by grepping the bundle, not the source. The source was already right; that is the whole trap:

D=devops/runtime/console/<sha>/browser
grep -rl "api/offlinekit/v1/exports" "$D"     # expect >= 1 hit in a chunk-*.js
grep -rl 'exportCenter:"/v1/exports"' "$D"    # expect ZERO
docker inspect stellaops-router-gateway \
  --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}' | grep wwwroot

On this estate 2026-09-14 those three checks read: 4 hits, 0 hits, and the mount resolving to devops/runtime/console/3b978a02a8aec5e92814d677203e978a06d7be25/browser — i.e. the rebuild landed and the pin moved with it.

Old bundle directories are not swept automatically. Six exist under devops/runtime/console/ on this estate. Keep at least the previously-pinned one as the rollback; the rest are reclaimable disk.

Release bundle images

The release bundle is built from the same source through a different path, and its start order is not operator-driven:

Cross-references