Healthcheck honesty fix — 2026-05-12

Sprint: SPRINT_20260512_015_DevOps_healthcheck_path_alignment Commit: df3ed409e — Fix lying healthchecks + flip defaults to /healthz + per-svc routes

This release closes the highest-impact finding from the 2026-05 orchestration cycle: every container healthcheck written before df3ed409e was reporting “healthy” without actually checking HTTP correctness. Operators reading prior “containers healthy” reports should re-evaluate them with this context. The fix is mandatory; rebuild images to propagate the corrected probe to running containers.

Operator impact (read this first)

Prior “containers healthy” reports were honest about TCP listen, but did NOT verify HTTP correctness. A service that bound its port but returned 404, 500, or never mapped a /health* route at all would report healthy in docker ps and docker inspect. Any operator runbook, smoke check, or CI gate that relies on Docker’s Status: healthy for the stellaops/<service>:* images is affected.

Concretely, this masked the following pre-existing latent bugs that SPRINT_20260512_015’s per-service survey then forced into the open:

Discovery

While preparing the 2026-05 release artifacts, the orchestration team inspected devops/docker/healthcheck.sh and noticed that the bash /dev/tcp fallback path (taken when curl and wget are unavailable — which is every stellaops image, none of them ship either tool) established a TCP connection and exited 0 without parsing the HTTP status line. A targeted probe via a curlimages/curl:latest sidecar attached to the stellaops docker network then surfaced the 10 services with no working /health* route and the dozen-plus services with a route under a non-default path.

Fix

devops/docker/healthcheck.sh was rewritten so the /dev/tcp fallback now:

  1. Issues a GET ${PATH} HTTP/1.0 request over the raw socket.
  2. Parses the response status line.
  3. Exits 0 only when the status line is a non-empty 2xx or 3xx within ${HEALTH_TIMEOUT} seconds.
  4. Exits 1 on any 4xx/5xx, on connection refusal, on timeout, or on an empty/garbled response.

Defaults were flipped to /healthz + /readyz to match the dotnet baseline and the strict majority of running services. 14 services received explicit LIVENESS_PATH / READINESS_PATH compose overrides that match the routes they actually map; 8 workers inherit /health/liveness + /health/readiness via the new &worker-health-env YAML anchor. 10 services with no /health* route were placed on a temporary TCP-listen-only probe via &healthcheck-port-listen while the app-side MapHealthChecks follow-ups are tracked under SPRINT_20260512_015’s *_HEALTH-01 BLOCKED tasks.

The retired C:/temp/policy-engine-health-override.yml was folded into the canonical compose entry; the temp file is no longer required.

Action required

Rebuild and recreate

The script lives in the runtime image layer, so a docker compose pull --no-build is not sufficient. To propagate the honest probe:

# Rebuild every service image so the corrected healthcheck.sh ships
docker compose -f devops/compose/docker-compose.stella-ops.yml \
  -f devops/compose/docker-compose.stella-services.yml build

# Recreate containers
docker compose -f devops/compose/docker-compose.stella-ops.yml \
  -f devops/compose/docker-compose.stella-services.yml up -d

The previous orchestration cycle verified 55/55 containers healthy post-recreate with --no-build because only compose env was changed; a full image rebuild is now required for the script fix itself to land in the runtime layer.

Re-evaluate prior smoke checks

If your CI pipeline, deployment runbook, or release gate accepts docker inspect ... healthy as truth — re-run those checks against the rebuilt images. Any service that flips from healthy to unhealthy was already broken; the new probe is only telling the truth.

Watch for the 10 known-broken services

The following services are pinned to a TCP-listen-only probe until their *_HEALTH-01 BLOCKED follow-up ships:

ServiceSprint follow-up
signerSIGNER_HEALTH-01 (DONE in SPRINT_20260512_023)
workflowWORKFLOW_HEALTH-01 (DONE in SPRINT_20260512_023)
unknowns-webUNKNOWNS_HEALTH-01 (DONE in SPRINT_20260512_023, runtime blocker remains)
opsmemory-webOPSMEMORY_HEALTH-01 (DONE in SPRINT_20260512_023)
riskengine-webRISKENGINE_HEALTH-01 (DONE in SPRINT_20260512_023)
notify-webNOTIFY_HEALTH-01 (pending)
airgap-controllerAIRGAP_HEALTH-01 (pending)
export-webEXPORT_HEALTH-01 (pending)
evidence-locker-webEVIDENCE_HEALTH-01 (pending)
attestor-tileproxyTILEPROXY_HEALTH-01 (DONE in SPRINT_20260512_022)

Five of the eleven have landed AddHealthChecks() + MapHealthChecks since the discovery; the remaining six are queued behind the same pattern. The TCP-listen probe is not equivalent to an HTTP probe — treat any of these services with extra scrutiny until the per-service follow-up closes.

Follow-up sprints filed

Files of interest