Image Rebuild Order Runbook

Audience

Operators and autonomous agents who need to rebuild one or more Stella Ops service images in a running compose stack. Specifically: anyone who has just changed code in a shared library (auth, router, plugin host, persistence) and wants the running stack to pick up the change without manual debugging of “why does the new image still behave like the old one.”

This runbook collects three independent gotchas that multiple 2026-05-29 sprints (046, 049, 050, 055, 060, 066) rediscovered ad-hoc. The cost of each rediscovery is roughly 30-90 minutes of agent or operator time; documenting them here is the cheapest possible mitigation.

Prerequisites

When to use this runbook

Operator situationAction
Single-service code change (no shared lib touched, no auth touched)Targeted rebuild of just that service + up -d --force-recreate <service>
Shared lib change (e.g. src/__Libraries/StellaOps.Auth.*, src/Router/__Libraries/)Full ordered rebuild per the order in the next section
Authority schema or signing-key changeAuthority FIRST, then all consumers (because consumers cache the JWKS validation keys in-process at startup)
Release Orchestrator execution bundle-only changeRun devops/build/package-execution-plugins.ps1 for the affected profile and remount/recreate agent-core; do not rebuild agent-core unless host runtime code, Dockerfile tools, or shared Plugin Host code changed
Plugin host source change (src/Plugin/StellaOps.Plugin.Host/)Plugin host + all plugin-consuming services (release-orchestrator, agent-core, scanner-worker) — together so the loader contract matches across host and consumers
Compose file or env-file change (no image change)docker compose up -d is sufficient — but verify with docker inspect because compose has been observed silently no-op’ing layered FS swaps (see Caveat 1 below)

Quick reference: the canonical rebuild order

When a change touches auth, router, or any shared library that crosses service boundaries, the canonical rebuild + restart order is:

  1. authority— JWT issuer / OIDC. Must be ready before any consumer restarts, otherwise consumers cache the OLD JWKS at boot.
  2. scanner-worker— registers per-tenant token handlers (Sprint 048 / Sprint 051). Must boot AFTER authority so the per-tenant client credentials flow against a current issuer.
  3. integrations-web— owns the registry credential resolver chain (Sprint 027 / Sprint 046). Must boot AFTER scanner-worker so the scanner-worker’s IntegrationAwareRegistryCredentialProvider (per-process cache) is keyed against the same integration-row state.
  4. Other consumer services (router-gateway, release-orchestrator, findings, etc.) in any order, but AFTER the three above.

This ordering exists because Stella’s bearer-token validation, integration catalog, and per-process credential caches all warm up at container startup; rebooting a downstream consumer before its upstream is ready bakes a stale view into the in-process state until the next restart.

Procedure

Step 1 — Stop everything that depends on the changed surface

For an auth-affecting change:

docker compose -f devops/compose/docker-compose.stella-ops.yml stop scanner-worker integrations-web router-gateway release-orchestrator findings

For a shared library that affects MOST services, prefer a full stack stop so no service holds a stale DLL via Docker’s overlay filesystem mount:

docker compose -f devops/compose/docker-compose.stella-ops.yml down --remove-orphans

--remove-orphans clears the named containers so the next up re-creates them rather than re-attaching to a stopped one with the old image digest pinned in metadata.

Step 2 — Rebuild images with --no-cache

For a targeted rebuild:

& 'C:\Program Files\Git\bin\bash.exe' -lc "cd '/c/dev/New folder/git.stella-ops.org' && SERVICES=authority,scanner-worker,integrations-web ./devops/docker/build-all.sh"

For a full rebuild (all 60+ services, ~20-40 min depending on host):

& 'C:\Program Files\Git\bin\bash.exe' -lc "cd '/c/dev/New folder/git.stella-ops.org' && ./devops/docker/build-all.sh"

For PowerShell-only rebuilds, build-all.ps1 -NoCache remains available, but operators must create a rollback tag manually first if they need the snapshot guarantee. Docker’s per-layer cache key is hash-of-context; a small change inside src/__Libraries/StellaOps.Auth.* may not change the layer key in a way Docker recognises, and the cached layer with the old compiled assembly can win. The cost of a cold rebuild is rebuild time; the cost of stale layers is hours of “why does the new code not run.”

For agent-core specifically (Sprint 066 caveat), use:

& 'C:\Program Files\Git\bin\bash.exe' -lc "cd '/c/dev/New folder/git.stella-ops.org' && SERVICES=agent-core ./devops/docker/build-all.sh"

— and confirm the new digest with docker images stellaops/agent-core --no-trunc.

Execution plugin bundle updates are separate from agent-core image rebuilds. For local/offline-kit evidence, build signed bundle profiles first:

powershell -NoProfile -ExecutionPolicy Bypass -File .\devops\build\package-execution-plugins.ps1 -Profile default
powershell -NoProfile -ExecutionPolicy Bypass -File .\devops\build\package-execution-plugins.ps1 -Profile optional-build-script

Then render the profile-gated agent overlay through the compose wrapper:

$env:COMPOSE_PROFILES = 'agents'
$env:COMPOSE_EXTRA_FILES = 'docker-compose.plugins.execution.yml'
.\devops\compose\scripts\compose-cli.ps1 config --format json

The agent-core image should contain the host runtime, /app/tools/StellaOps.Plugin.Host, and required CLIs only. Execution payload files such as plugin.json, plugin.signature.json, StellaOps.Agent.BuildDocker.dll, StellaOps.Agent.BuildScript.dll, StellaOps.Agent.DeployCompose.dll, and StellaOps.Agent.DeployNative.dll must come from read-only mounted bundle roots. When a local image exists, verify that with:

powershell -NoProfile -ExecutionPolicy Bypass -File .\devops\build\audit-pluginized-image.ps1 -Image stellaops/agent-core:dev -Service agent-core -FailOnPluginRootPayload

Step 3 — Restart in the canonical order with --force-recreate

docker compose -f devops/compose/docker-compose.stella-ops.yml up -d --force-recreate authority
# Wait for authority to be healthy (next step) before bringing the next service up
docker compose -f devops/compose/docker-compose.stella-ops.yml up -d --force-recreate scanner-worker
docker compose -f devops/compose/docker-compose.stella-ops.yml up -d --force-recreate integrations-web
docker compose -f devops/compose/docker-compose.stella-ops.yml up -d --force-recreate

The final up -d --force-recreate (with no service argument) brings up everything else, applying --force-recreate to any service whose tag actually changed. This is intentionally idempotent.

--force-recreate is REQUIRED because container restart alone does NOT swap the layered filesystem to a freshly-tagged image — docker compose restart only stops and starts the existing container against its pinned image digest. The newly-baked image is then inert until the container is destroyed and re-created. This was rediscovered by the Sprint 066 S066-001 agent (commit ef5120dd4c) when the rebuilt agent-core image with docker-buildx baked in appeared not to ship the new binary until --force-recreate ran.

Step 4 — Verify each service picked up the new image

docker inspect stellaops-authority --format '{{.Image}}'
docker inspect stellaops-scanner-worker --format '{{.Image}}'
docker inspect stellaops-integrations-web --format '{{.Image}}'

Cross-check the returned image SHA against docker images stellaops/authority --no-trunc --format '{{.ID}}'. If they match the newly-baked digest, the swap landed. If not, repeat Step 3 with the explicit service name AND --force-recreate.

For the agent-core image specifically, confirm the bundled AGENT_CORE_CAPABILITIES env-file is a list of 10 capability strings (Sprint 066 caveat):

docker exec stellaops-agent-core printenv AGENT_CORE_CAPABILITIES

Empty or short lists indicate the env-file was not picked up at boot and the agent will fail to register its capability set with the orchestrator.

Step 5 — Smoke check

Run the gateway sentinel curl per tenant-onboarding.md (or a simpler curl -k https://stella-ops.local/.well-known/openid-configuration against authority) to confirm bearer-token issuance still works against the rebuilt authority.

Caveats (the costly rediscoveries)

Caveat 1 — docker compose restart does NOT pick up new images

Rediscovered by Sprint 066 S066-001 (commit ef5120dd4c) — the rebuilt agent-core image with docker-buildx baked in appeared inert until --force-recreate ran. restart is for picking up env-var changes or applying a graceful in-process refresh; it does NOT swap the overlay filesystem to a freshly-tagged image of the same name. Always use up -d --force-recreate <service> after a rebuild.

Caveat 2 — AGENT_CORE_CAPABILITIES env-file 10-capability requirement

Rediscovered by Sprint 050 (commit referenced in SPRINT_20260529_050) — docker-compose.lab-plugin-bundles.yml set AGENT_CORE_CAPABILITIES to a bare deploy.compose string. The compose plugin’s manifest declares deploy.compose.{pull,preflight,up,...} as 10 separate capabilities; the orchestrator’s catalog resolves by alias namespace, so the bare string fails silently — the plugin is discovered, signature-verified, and trust-mapped, but never instantiated because no capability matched. Always pass the full capability list (10 entries for the standard compose deploy plugin) in AGENT_CORE_CAPABILITIES.

Caveat 3 — In-process auth state caches stale token-validation keys

Rediscovered by Sprint 060 S060-001 (PluginTrustRoot regression discovery, commit 5df04f6b02 was the trigger). Authority issues JWTs signed with a per-installation key. Each consumer service fetches the JWKS at startup and caches it in-process. If you rebuild + restart authority, then rebuild a consumer that boots BEFORE authority is fully healthy, the consumer caches the OLD JWKS (or fails to bootstrap if authority is fully down). Symptoms: 401 Unauthorized on previously-working endpoints with a signature failed reason in the audit log. Resolution: stop the consumer, wait for authority /health to return 200, restart the consumer.

Caveat 4 — --no-cache is required after a shared-library change

Rediscovered by Sprint 049 (loader-side DI activation pipeline work) and earlier. Docker’s per-layer cache key may not change in a way Docker recognises when a small edit lands inside a transitively-included shared assembly. The cached layer with the OLD compiled DLL wins, and the rebuild silently produces the same image. Always pass -NoCache after any change inside src/__Libraries/, src/Router/__Libraries/, src/Plugin/StellaOps.Plugin.Host/, or src/Authority/__Libraries/StellaOps.Auth.*.

Cross-references