AdvisoryAI — building and deploying the staged consolidated stack (AAI-8)

Sprint: SPRINT_20260722_013 (AAI-8 stages; AAI-9 cuts over; AAI-10 deletes). Overlay: devops/compose/docker-compose.advisoryai.yml — deliberately outside the default config_files list. Publish keys: advisoryai-web, advisoryai-worker in devops/docker/build-service-publish.sh (AAI-9 gate 10). They replace the two legacy deployable keys; they do not coexist with them. Route swap package: tools/scripts/aai5-advisoryai-route-swap.json plus tools/scripts/aai5-swap-advisoryai-routes.ps1. The former standalone AAI-5 runbook is frozen history; the active window order is the AAI-9 gate list in the sprint, not a nonexistent docs/runbooks/advisoryai-cutover-* page.

Superseded in part, 2026-09-07. The window happened: AAI-9 moved the data on 2026-09-04 and AAI-10 deleted the three predecessor compose keys and retired the OpsMemory host. Read what follows as the staged-deploy procedure it was written as; every statement below about the predecessors still serving describes the estate BEFORE that date.

The three keys this family replaced — advisory-ai-web, advisory-ai-worker and opsmemory-web — kept serving until the AAI-9/AAI-10 window. Nothing here changes the running stack.

STOP — this overlay is not window-ready on its own. AAI-9 gates 1–4, 9 and 10 closed the former persistence, route, worker and OpsMemory source seams. Current source now converges the consolidated database, registers the durable PostgresEvidencePackStore, and closes F-AAI5-17 at 7d30dbf4ea79cba79569413839237c6ac84e3347: Production resolves an UnconfiguredEvidencePackSigner, so unsigned operations can reach storage and only explicit sign/verify calls fail closed. A Production/own-DSN regression with local harness fakes disabled proves that source contract. Clean successor images from 8ca1fac2741b8a5d5e23de16112eff812559e003 were rebuilt and the isolated direct-host rehearsal passed on 2026-08-27; the receipt below closes that prerequisite. Router stayed disabled in the safe scratch topology, so only AAI-9’s actual gateway window can exercise F-AAI5-6’s ReverseProxy-to-Microservice route. Do not convert the direct-host result into a gateway-route or live-estate result.

Why the publish keys are a rename, not duplicates

Findings, JobEngine and the vulnerability hub all registered their consolidated deployable keys in build-service-publish.sh and docs/architecture/build-boundary/ownership-manifest.json at their S8, and for them it was free: their consolidated hosts were born new, with zero build-boundary violation pairs, so registering the keys added four keys and no register rows.

This family’s hosts are continuation hosts (D-AAI3-1). advisoryai-web and the live advisory-ai-web build from the same project (src/AdvisoryAI/StellaOps.AdvisoryAI.WebService); the worker has the same relationship. A second deployable key over either seed would describe no new service and would count the same source graph twice.

AAI-9 gate 10 therefore replaced both names atomically in the publish helper and ownership manifest. New builds produce stellaops/advisoryai-web:* and stellaops/advisoryai-worker:*. The helper’s third fields retain stellaops/advisory-ai-*:dev only as the explicit legacy-base escape hatch; they do not name the output images. The two predecessor images are temporarily listed as compose-only rollback inputs because the canonical stack still runs them until the window/AAI-10. They are not deployable-key duplicates. AdvisoryAiConsolidationConformanceTests.ConsolidatedDeployableKeys_ReplaceLegacyKeys_AndTheOverlayBuildsSuccessorImages pins all three facts.

What a freshly built staged stack serves (source-ready, not live)

advisoryai-web continues the live AdvisoryAI host under a new compose name. The current source was built and booted under a disposable local tag on 2026-08-26, then that tag was removed; it has not been deployed to the standing estate. Once built for the window, it preserves everything advisory-ai-web serves today and adds:

• GET /doctor/advisoryai-web/checks — doctor-check/v1, guarded by the ops.health scope (AAI-7).
• Native /api/advisoryai/v1/** aliases for every carried AdvisoryAI search/advisory/run/chat/evidence-pack route (AAI-9 gate 3). The legacy paths remain in the same image for rollback.

Two things it does not do:

1. It does not retire OpsMemory rollback material. Gate 9 now mounts all six carried OpsMemory endpoints and their native aliases in the continuation web host, but opsmemory-web, its schema and data remain intact through soak and AAI-10.
2. It does not register Doctor by default. The own DSN and already-reconciled Authority scope are prerequisites; enable registration only in the window and prove the Platform capability row.

The doctor surface and the owned-DSN gate

advisoryai-web binds its doctor data source from STELLAOPS_POSTGRES_ADVISORYAI_CONNECTION (or the equally specific Postgres:AdvisoryAI:ConnectionString) and from nothing else. It will not fall back to AdvisoryAI:Storage, Postgres:Concelier, or ConnectionStrings:Default. Current source uses that same owned DSN for the startup migrator, runtime stores, Knowledge Search, chat audit and OpsMemory.

Consequence: a Production successor without the owned DSN fails closed during runtime binding; it no longer starts in a legacy-storage compensation mode. Once the DSN resolves, the doctor checks and all domain stores observe the same database, and advisoryai-web is the sole startup-migration authority.

Registration is OFF, and this family has an extra trap

Doctor__Registration__Enabled defaults to false. Turning it on before this service’s Authority client holds platform:doctor:register produces a 403 loop against Platform — the failure the default-off posture exists to prevent.

The pre-window Authority reconciliation completed on 2026-08-22: the existing stellaops-advisory-ai-internal row gained only platform:doctor:register, and a fresh default-tenant token proved that scope plus the existing stellaops audience. Registration stayed off. At cutover, confirm this persisted state; do not repeat the grant as an unverified write or treat it as capability-row proof.

Two traps when the window flips it:

• Set the values, do not just declare them. The reference-adopter wiring guards Authority/ClientId/ClientSecret with ?? throw, which accepts an empty string, and compose passes unset variables through as ${VAR:-}. This host rejects blank as well as missing values and refuses to start (DoctorRegistration_FailsClosed_OnABlankCredential), so a failed start immediately after the flip means one of the three is empty.
• Do not give the doctor registrar its own client id — this host holds exactly ONE StellaOps auth client and AdvisoryAi:Authority already owns it (stellaops-advisory-ai-internal, which the five UnifiedSearch live adapters authenticate with). Leave Doctor__Registration__ClientId unset so the registrar rides that identity, and grant that client the platform:doctor:register scope. Setting a different id fails the host closed by design, because the alternative is silently repointing every live adapter at the doctor client and losing its advisory:read … grants. This trap is unique to this family: the sibling adopters were born-new hosts with no auth client to hijack.

Build the images

Use a unique scratch tag and the repository’s publish-then-local-image workflow. Despite the helper’s name, this path performs dotnet publish plus a local docker build; it does not push a registry image. Do not publish the scratch tag.

export AAI5_SCRATCH_ID="aai5scratch-$(date -u +%Y%m%d%H%M%S)-$$"
export ADVISORYAI_IMAGE_TAG="$AAI5_SCRATCH_ID"
IMAGE_TAG="$ADVISORYAI_IMAGE_TAG" STELLAOPS_PLUGIN_PACKAGING_MODE=mounted \
  bash devops/docker/build-service-publish.sh advisoryai-web advisoryai-worker

The doctor SDK is prune-sensitive. StellaOps.Doctor.Plugin.Abstractions.dll matches the image pruner’s *.Plugin.*.dll pattern by NAME and survives only because it is allowlisted in build-service-publish.sh (line ~370). If advisoryai-web — or, from now on, the live advisory-ai-web — ever crash-loops at boot with a FileNotFoundException for that assembly, the allowlist regressed; do not “fix” it by removing the doctor adoption.

Record the source commit and both resulting local image IDs before the scratch boot. AAI-5 acceptance is invalid if either image tag resolves to a predecessor image or a build from another commit.

Bring up an isolated scratch stack

The base family overlay is not isolated by -p: its fixed carried-volume names still resolve to the standing queue/plan/output volumes. The scratch overlay must be last. It replaces all five physical volume names, all four container names, every inherited network alias, and the physical network name with the same required unique prefix. It also pins Router transport off and requires a fresh scratch identity-envelope key.

Keep one shell open from preparation through teardown. Generate the two scratch credentials locally without printing them; load the existing development certificate password and an audited digest-pinned PostgreSQL image through their environment-variable names. Never source devops/compose/.env and never reuse a prefix beginning with stellaops or compose.

test -n "${AAI5_SCRATCH_ID:-}" || exit 1
case "$AAI5_SCRATCH_ID" in stellaops*|compose*) exit 1 ;; esac
export ADVISORYAI_SCRATCH_RESOURCE_PREFIX="$AAI5_SCRATCH_ID"
export ADVISORYAI_ROUTER_ENABLED=false
export ADVISORYAI_POSTGRES_PASSWORD="$(openssl rand -base64 36 | tr -d '\r\n')"
export STELLAOPS_IDENTITY_ENVELOPE_SIGNING_KEY="$(openssl rand -base64 48 | tr -d '\r\n')"
export STELLAOPS_POSTGRES_ADVISORYAI_CONNECTION="Host=${ADVISORYAI_SCRATCH_RESOURCE_PREFIX}-db;Port=5432;Database=stellaops_advisoryai;Username=advisoryai;Password=${ADVISORYAI_POSTGRES_PASSWORD};Maximum Pool Size=16"
: "${ADVISORYAI_IMAGE_TAG:?build the two successor images under the unique scratch tag first}"
: "${POSTGRES_IMAGE:?set an audited digest-pinned local PostgreSQL image}"
: "${KESTREL_CERT_PASSWORD:?load the development certificate password from its local secret source}"

python tools/scripts/validate/validate_runtime_posture.py --self-test
python tools/scripts/validate/validate_runtime_posture.py

cd devops/compose
docker compose -p "$AAI5_SCRATCH_ID" --env-file /dev/null \
  -f docker-compose.advisoryai.yml \
  -f docker-compose.advisoryai.scratch.yml config --quiet
docker compose -p "$AAI5_SCRATCH_ID" --env-file /dev/null \
  -f docker-compose.advisoryai.yml \
  -f docker-compose.advisoryai.scratch.yml up -d

The preflight is falsifiable in two layers:

• --env-file /dev/null. Compose loads devops/compose/.env by default and can carry the estate’s shared connection and secrets. The scratch overlay has required interpolation, so missing prefix, image tag, owned DSN, database password, certificate password, PostgreSQL image, or envelope key makes config --quiet fail before any resource exists.
• validate_runtime_posture.py --self-test mutates a synthetic scratch overlay to a known live physical volume name and succeeds only when the detector rejects it. The normal validator checks all five explicit last-file volume overrides, the prefixed network/containers/aliases, the required signing input, and the frame-envelope primary-key pairing.
• AdvisoryAiConsolidationConformanceTests renders both files with non-secret sentinels and asserts the exact physical names, no inherited live alias, no published port, the owned scratch DSN, Router__Enabled=false, and identical identity/frame signing scalars.
• ADVISORYAI_IMAGE_TAG is required. Record the tag, image IDs and source commit. A predecessor or unrelated local image is not acceptance evidence.
• The scratch cluster proves nothing about the ROLE boundary — see the next section. On a throwaway single-database cluster the rehearsal role owns everything by construction.

Teardown only the exact project and the five exact prefixed volumes. The residue checks fail if a project container, volume, or network survives; do not broaden cleanup to a prune or any stellaops-* resource.

docker compose -p "$AAI5_SCRATCH_ID" --env-file /dev/null \
  -f docker-compose.advisoryai.yml \
  -f docker-compose.advisoryai.scratch.yml down --volumes --remove-orphans

test -z "$(docker ps -aq --filter "label=com.docker.compose.project=$AAI5_SCRATCH_ID")"
for suffix in advisory-ai-queue advisory-ai-plans advisory-ai-outputs advisoryai-plugin-scratch postgres-data; do
  ! docker volume inspect "${ADVISORYAI_SCRATCH_RESOURCE_PREFIX}-${suffix}" >/dev/null 2>&1
done
! docker network inspect "${ADVISORYAI_SCRATCH_RESOURCE_PREFIX}-network" >/dev/null 2>&1

What the current scratch boot converges

The 2026-08-10 rehearsal measured the predecessor path: 26 legacy advisoryai tables, no opsmemory schema, and the old baseline ledger. That is historical evidence, not the current expectation. AAI-9 gates 1–2 subsequently retired the legacy migrator/private ledger and repointed all runtime persistence to the owned DSN.

Current source makes advisoryai-web the sole startup-migration authority. The 2026-08-26 isolated boot from be0cbd57e490afa6ec844641b59f11418f698b4e measured this non-vacuous result:

Verified-by: digest-pinned PostgreSQL 18.1 in isolated Compose project aai5scratch-be0cbd5-r1; information_schema.tables, to_regclass and advisoryai.schema_migrations queries before teardown, then the ledger query again after docker compose ... restart advisoryai-web. Do not copy the 2026-08-10 legacy numbers or the pre-002 24-table count into new acceptance evidence.

Direct-host forcing history

The same boot used a fresh scratch-only identity-envelope key, Router disabled, no published host port, and a local curl probe attached only to the scratch network. /health/ready returned 200. On /api/advisoryai/v1/evidence-packs, anonymous, correctly signed wrong-scope and bad-signature calls returned 401, 403 and 401. Correctly signed list and create calls both returned 500; the host log named:

AdvisoryAI production runtime binding for IEvidencePackSigner is not configured.

That measured run was not a database failure. Its source registered a throwing production signer factory, while EvidencePackService required that signer in its constructor for every endpoint, including unsigned list/create. The durable restart test enabled AdvisoryAI:RuntimeBindings:AllowLocalHarnessFakes=true, so it verified the store but could not falsify the Production composition gap.

The bounded source correction is now commit 7d30dbf4ea79cba79569413839237c6ac84e3347. Production registers UnconfiguredEvidencePackSigner; resolution succeeds, and only sign/verify use throws the existing named configuration exception. Verified-by: EvidencePack_ProductionComposition_AllowsUnsignedOperations_AndSigningFailsOnUse boots Production against a fresh own database with local fakes disabled. Its parent red returned list=500; corrected source returns list=200, persists an unsigned pack and rejects both sign and verify (1 selected / 933 suite). At that point the TestServer proof did not replace a clean-image/direct-host rehearsal. The later 2026-08-27 receipt below now supplies that evidence.

2026-08-27 reboot stop — Docker unavailable; no runtime result

At detached source commit 8ca1fac2741b8a5d5e23de16112eff812559e003, the targeted Production-composition regression restored, built and selected 1 of the 933-case suite, but its PostgresFixture constructor threw DotNet.Testcontainers.Builders.DockerUnavailableException before the test body ran. Testcontainers reported both unix:///var/run/docker.sock and npipe://./pipe/docker_engine unavailable. This run completed zero product assertions and neither confirms nor reverses the earlier source result.

Docker Desktop was not started: an unattended start may wake restart-policy standing-estate containers, outside this scratch-only lane. Therefore no image, Compose project, container, network, volume, database or endpoint probe was created, and no teardown was required. The daemon-free checks do not substitute for the missing direct-host forcing:

• AdvisoryAiConsolidationConformanceTests: 12 selected / 12 passed;
• runtime-posture detector: self-test 1/1, unit suite 22/22, normal scan exit 0 with 66 existing warnings;
• Release dotnet publish: exit 0 for both successor hosts, with the web and worker DLLs and Windows apphosts present. These are source-graph outputs, not images and not runtime evidence.

At that stop, the authorized next step was an owner-attended Docker Desktop session or an explicitly isolated daemon that could not wake the standing estate. The smallest daemon-backed source forcing retry was:

pwsh ./tools/scripts/test-targeted-xunit.ps1 `
  -Project src/AdvisoryAI/__Tests/StellaOps.AdvisoryAI.Tests/StellaOps.AdvisoryAI.Tests.csproj `
  -Method "*EvidencePack_ProductionComposition_AllowsUnsignedOperations_AndSigningFailsOnUse*" `
  -BuildProjectReferences -Restore

The resumed run completed that fixture green and did not reuse the daemon-free publish outputs as image evidence. It set unique AAI5_SCRATCH_ID and ADVISORYAI_IMAGE_TAG values, then made the first non-substitutable runtime step:

IMAGE_TAG="$ADVISORYAI_IMAGE_TAG" STELLAOPS_PLUGIN_PACKAGING_MODE=mounted \
  bash devops/docker/build-service-publish.sh advisoryai-web advisoryai-worker

The receipt below records both image IDs plus scratch preparation, render, forcing, catalog/restart checks and exact teardown. Router remained disabled and no image was published.

2026-08-27 resumed clean-image/direct-host result

Docker became reachable externally after the stop above; this lane did not start it and never inspected or counted standing resources. From detached clean source 8ca1fac2741b8a5d5e23de16112eff812559e003, the publish helper built only the two local successor images, with no registry push:

Both embedded build-info records named that commit and worktree=clean. The last-file render for exact project/prefix aai5scratch-8ca1fac-r2-061251 contained four prefixed services, one prefixed network and five prefixed volumes, with Router false and no published ports. It used the already local, digest-pinned PostgreSQL image postgres@sha256:1090bc3a8ccfb0b55f78a494d76f8d603434f7e4553543d6e807bc7bd6bbd17f. PostgreSQL, web and worker became healthy; data-init exited 0.

The first secret-silent forcing run measured:

Verify needs a stored signed representation before it invokes the signer. The rehearsal therefore attached, in the scratch database only, a syntactically valid DSSE wrapper whose pack member was the JSON-identical stored pack, then called the native verify endpoint. EvidencePackService.VerifyAsync invoked the real Production signer first and produced the named failure; no fake signer or successful verification was introduced. An independent direction re-force of the same secret-silent sequence created a second random pack and again passed every status and named-error assertion. After its exact web restart, both packs and both run links persisted.

Catalog queries before and after restart reported 27 advisoryai tables and exactly one opsmemory table; advisoryai.schema_migrations contained exactly 001_advisoryai_consolidated_baseline.sql and 002_advisoryai_evidence_pack_store.sql once each; neither opsmemory.schema_migrations nor advisoryai.__migration_history existed; the final direction run measured 2 Evidence Packs, 2 run links and 0 OpsMemory decisions. The ledger and catalog were unchanged by restart.

Focused verification at the same source passed the Production composition body 1 selected / 933 suite and deleted its Testcontainers PostgreSQL resource; AdvisoryAiConsolidationConformanceTests passed 12/12. The daemon-free runtime-posture detector evidence from the earlier stop remains 1/1 self-test, 22/22 unit suite and exit 0 with 66 existing warnings. The two earlier Release publishes also remained source-graph evidence only.

Teardown used the exact project with down --volumes --remove-orphans, which removed all four prefixed containers, the one network and all five volumes. Exact-name container, volume and network inspections then returned rc=1; both unique image tags were removed and exact image inspection returned rc=1. The temporary publish outputs and forcing scripts were removed. No standing route, container, database, image or Compose project was inspected or changed.

This closes the clean-image/direct-host prerequisite only. AAI-5 remains pending and its Router-named forcing criterion remains unticked until AAI-9 exercises the consolidated route in the owner-attended window.

The volumes are carried, not recreated

advisory-ai-queue, advisory-ai-plans and advisory-ai-outputs are the same three named volumes the legacy keys mount, so the window keeps queued work, the plan cache and rendered outputs across the swap. In the base overlay their physical names are fixed and ignore -p; that is required for the window and unsafe for rehearsal. The last-file scratch overlay replaces those three, the writable plugin scratch, and the PostgreSQL data volume with five names under ADVISORYAI_SCRATCH_RESOURCE_PREFIX.

The advisoryai-data-init step exists for the same reason it exists in the live file: the volumes must be chowned to uid 10001 before either role starts.

The database owner role — decide this before AAI-9, not during

The role IS the boundary — there is no container around this database. Per the owner clarification of 2026-08-10, stellaops_advisoryai is a logical database on the shared control-plane cluster, so nothing but the role stops another service’s role from reading AdvisoryAI’s tables. AAI-9 creates it with:

CREATE ROLE advisoryai LOGIN PASSWORD '<from ADVISORYAI_POSTGRES_PASSWORD>';
CREATE DATABASE stellaops_advisoryai OWNER advisoryai;
REVOKE CONNECT ON DATABASE stellaops_advisoryai FROM PUBLIC;
GRANT  CONNECT ON DATABASE stellaops_advisoryai TO advisoryai;

The REVOKE is the load-bearing line and it is the one that gets skipped, because PostgreSQL grants CONNECT to PUBLIC on every new database by default: without it every other service role on the shared cluster still reaches these tables and the move buys isolation on paper only. Symmetrically, advisoryai must get CONNECT on no other service’s database — the cheap way to find a cross-database read nobody inventoried is to watch what starts failing after that revocation, which is what AAI-9’s soak measures. Scanner precedent: e01c4a43ce.

Beyond the role, the schema-level posture:

This family’s baseline creates no roles and enables no row-level security, while 8+ of its tables carry a NOT NULL tenant_id. Both sibling families are the other way round (Findings ships FORCE RLS; JobEngine ships RLS plus a scheduler_admin role), so do not carry their grant step over unexamined:

• there is no baseline role to grant — AAI-9 creates the owner role itself;
• tenant isolation is application-enforced only. src/AdvisoryAI sets no app.tenant_id session GUC anywhere, so enabling RLS is a posture decision with an application cost, not a one-line edit to a still-unapplied baseline;
• GRANT … ON ALL TABLES IN SCHEMA is a one-time snapshot: a table created after the grant is ungranted unless the step is paired with ALTER DEFAULT PRIVILEGES. Measured — this family has zero partitioned tables, so the partition-leaf variant of that hazard does not apply.

All of the above is measured from the live catalog by StellaOps.AdvisoryAI.Persistence.Tests.AdvisoryAiRoleGrantTests, so it fails loudly if a later migration changes any of it.

What AAI-9 still owes after the direct-host rehearsal

1. Build and publish only the two successor keys for the owner-attended window, record their source/image identity, and preserve exact predecessor image IDs/tags as rollback inputs. The removed scratch tags above are rehearsal evidence, not deployable window inputs.
2. Exercise the realistically sized evidence-pack export through the actual Router Microservice transport, not only direct HTTP to advisoryai-web. Router is deliberately disabled in the safe scratch overlay, so this remains an AAI-9 routing prerequisite and is the empirical F-AAI5-6 ceiling test.
3. In the owner-attended window, create/harden the database role, copy the 25 domain tables with parity, prove sibling-database denial, enable Doctor, apply the route/Console swaps, run the Router forcing set and soak, then flip the ownership matrix. Preserve predecessor containers/schemas until AAI-10’s separate retirement gate.