AdvisoryAI AAI-9 own-database data move and rollback
Audience: AdvisoryAI implementer, database operator, live-deploy operator, rollback owner, and the owner approving the SPRINT_20260722_013 AAI-9 maintenance window.
Purpose: converge the empty stellaops_advisoryai target from the successor’s own startup migrations, freeze the three predecessor writers, copy only the 25 carried domain tables, prove exact row-count and content-digest parity, repoint the family onto the consolidated overlay, and preserve a tested rollback path before any route, flag, grant, or source-retirement action.
Window execution plan (timed sequence, preflight NO-GO list, point of no return, per-phase rollback, forcing functions, roles, durations): advisoryai-aai9-window-plan.md, prepared 2026-09-04. It also records five verified defects in the steps below — the step-2 volume names, the host pg_restore calls in steps 4 and 7, the step-3 isolation probe’s coverage failure, the missing live Compose array for steps 3 and 10, and --no-deps skipping advisoryai-data-init — with their fixes. Read it before executing this runbook.
This is an operator procedure, not authorization to run it. It was prepared from source and from read-only measurements against the live estate on 2026-08-28; it is not an execution receipt. Stop at the first failed assertion. Do not improvise a whole-schema restore, copy any migration ledger, or unfreeze a writer while authority is ambiguous.
Preflight measurements this runbook is built on (Phase A, 2026-08-28). Every one is re-runnable; each step below names the command that falsifies it.
| Finding | Verified by |
|---|---|
| pgvector is not available on the server at all and installs nowhere | SELECT * FROM pg_available_extensions WHERE name IN ('vector','pgvector','vectors') returns zero rows on PostgreSQL 18.1 |
No vector-typed column exists on either side; kb_chunk.embedding is real[] (_float4) | information_schema.columns on advisoryai.kb_chunk, both databases |
| Column definitions of all 25 carried tables are identical between live and a fresh baseline | 260/260 rows of the column probe in step 9, empty diff |
| Constraints are identical: 201/201 | constraint probe in step 9, empty diff |
Indexes are identical, 85/85 with an empty diff, once step 3’s pg_trgm pin is applied; without it the same probe carries two trigram opclass divergences | 85/85 index probe run both ways (finding D1) |
Column ordinal order differs on kb_chunk and search_events (finding D2) | ordinal probe in step 9 |
| Zero sequences and zero identity/serial columns on both sides | pg_class.relkind='S' = 0, and is_identity='YES' OR column_default LIKE 'nextval%' = 0, both databases |
| Carried row total is 13,238 across five non-empty tables | per-table count(*) sweep, step 3 |
No FORCE RLS anywhere: advisoryai 0 forced / 0 enabled of 26, opsmemory 0/0 of 2, zero pg_policy rows — so the NTF-9 crash-loop class does not apply here (finding D6) | relforcerowsecurity/relrowsecurity survey + pg_policy count, step 3 |
A superuser session hides a FORCE-RLS failure; SET ROLE and a real service-role login both expose it | three-way sandbox red-proof, finding D6 |
Four hosted services write carried tables on start, all defaulting true in code and all now explicitly off in the overlay (F-AAI9-11) | x-advisoryai-carried-table-writers; pinned by StagedComposeRender_DefaultsEveryCarriedTableWriterOff (red-proofed) and asserted end-to-end by step 3’s tables_checked=27 rows=0 |
Non-negotiable boundaries
- Source schemas
advisoryaiandopsmemoryinstellaops_platformstay intact through rollback retention. This runbook drops no schema, table, database, role, volume, or container. Their destructive retirement is AAI-10, under a separately approved window. - Never copy a migration ledger or a ledger row. Three ledger relations exist and all three are excluded by name:
advisoryai.schema_migrations(predecessor ledger),advisoryai.__migration_history(the private ledgerPostgresKnowledgeSearchStore.EnsureSchemaAsyncused to create — F-AAI3-7), andopsmemory.schema_migrations(the legacy per-schema ledger). The converged target owns a fresh successor ledger with its own two rows. advisoryai-webis the family’s sole migration authority.Program.csregistersAddConsolidatedAdvisoryAiPersistencebeforeAddAdvisoryAiCoreadds either index hosted service, gated onTryResolveConnectionString. The worker must never be given a second migrator, and no legacy image may be pointed at the target: it carries the predecessor migration lineage.- The converge boot runs with all four carried-table writers OFF. Every one of them defaults to
trueinKnowledgeSearchOptions, and the overlay now sets all four explicitly false (x-advisoryai-carried-table-writers, F-AAI9-11), pinned byStagedComposeRender_DefaultsEveryCarriedTableWriterOffwith a red-proof. Step 3 asserts the render and the outcome — zero rows outside the ledger — before the dump. Step 10 turns them back on; they are product features, and leaving them off past the cutover is its own defect. - The target stays reachable only by its own role.
advisoryaihasCONNECT=falseonstellaops_platformandstellaops_findings(tools/scripts/deploy/postgres/probe-database-isolation.shreported PASS on 2026-08-28). The copy is executed by the cluster operator role against both databases; never widen the service role to make a copy convenient. - The consolidated overlay’s network alias
advisoryai.stella-ops.localis the same aliasadvisory-ai-webpublishes indevops/compose/docker-compose.stella-services.yml. The two files must never be composed together while both services run: the window stops the legacy hosts first (devops/compose/docker-compose.advisoryai.yml:192-201). - Run from the repository root in Bash. Receipts go under the gitignored
tmp/aai9-<window-id>/. Never put a password, token, connection string, or decrypted secret in a command transcript or a checked-in file. - Converge and verify the target under the
advisoryairole, never asstellaops. A superuser session bypasses row-level security unconditionally and would hand the window a false green — the NTF-9 failure mode (finding D6). The one deliberate exception is the step-8 restore. - A count-only comparison is insufficient. Every copied table needs count plus canonical row SHA-256 parity.
kb_chunk(10,154 rows) is the only table on this programme that carries real volume; its digest is the load-bearing one. - Any parity failure happens before source retirement. Keep the writer freeze, do not change routes or grants, and enter rollback.
Source-derived copy contract
The 25 carried domain tables are the intersection of the live predecessor schemas and the tables created by the successor’s 001_advisoryai_consolidated_baseline.sql. Row counts are the 2026-08-28 read-only measurement and must be re-measured at the window boundary (step 3); they are recorded here so a drift is visible rather than assumed.
# advisoryai schema — 24 tables (trailing comment = 2026-08-28 row count)
AAI9_COPY_ADVISORYAI=(
ai_claim_attestations # 0
ai_consents # 0
ai_run_attestations # 0
api_operation # 2190
api_spec # 1
chat_evidence_links # 0
chat_messages # 0
chat_policy_decisions # 0
chat_sessions # 0
chat_tool_invocations # 0
conversations # 0
doctor_search_projection # 186
entity_alias # 0
kb_chunk # 10154
kb_doc # 707
runtime_chat_settings_overrides # 0
runtime_explanations # 0
runtime_policy_intents # 0
runtime_runs # 0
search_events # 0
search_feedback # 0
search_history # 0
search_quality_alerts # 0
turns # 0
)
# opsmemory schema — 1 table
AAI9_COPY_OPSMEMORY=(
decisions # 0
)
# Converged empty on the target by forward migration 002; never populated by the copy.
AAI9_TARGET_ONLY=(
advisoryai.evidence_packs
advisoryai.evidence_pack_run_links
)
Carried total: 13,238 rows across five non-empty tables — a COPY-TIME number, and the five tables carrying all of that volume are a DERIVED index that step 10 re-derives. Both halves of that sentence have to travel together; separating them is the AAI-12 defect. Split explicitly:
# DERIVED — rebuilt from the repository corpus by KnowledgeAutoIndexOnStartup on the next boot.
# These five are exactly the five non-empty carried tables, so "13,238 rows migrated and verified"
# describes a state that holds from step 8 to step 10c and no longer holds afterwards.
AAI9_DERIVED=(
advisoryai.api_operation
advisoryai.api_spec
advisoryai.doctor_search_projection
advisoryai.kb_chunk
advisoryai.kb_doc
)
# INVARIANT — the remaining 20 carried tables. No writer re-derives these, so their parity is a
# steady-state property and step 10d re-asserts it AFTER the activation.
Two numbers, never one: rows copied and verified at the fence (13,238 as of the 2026-08-28 measurement, re-measured at step 3) and rows resident after activation (measured at step 10d; the executed window recorded 12,091 — kb_chunk 10154 → 9055 and kb_doc 707 → 659, the rest re-derived at identical counts). Report both or neither. A single figure is either stale or misleading depending on when it is read.
Explicit exclusions:
advisoryai.schema_migrations— predecessor ledger; the target converges its own.advisoryai.__migration_history— the retired private ledger (F-AAI3-7). Current source no longer creates it (AdvisoryAiRuntimeBindingGuardTests.cs:428-429pins its absence from the store); it survives on the live database only because the running image predates that change. It must not appear on the target.opsmemory.schema_migrations— legacy per-schema ledger, retired withopsmemory-webat AAI-10.- the two target-only Evidence Pack tables above.
opsmemory contributes zero business rows. Its only domain table, decisions, is empty, and its ledger row is excluded — so the recorded “two-schema move” transfers one schema’s data. decisions is still copied (as an empty table) so the contract stays uniform and a late write before the freeze cannot be silently dropped.
F-AAI3-8 is closed and not reproducible in this window. The duplicate search_feedback creation lived in the collapsed legacy baseline; StellaOps.AdvisoryAI.csproj now embeds no migration SQL at all (its only EmbeddedResource is synthesis-system-prompt.txt), and both duplicate definitions survive only under src/AdvisoryAI/StellaOps.AdvisoryAI/Storage/Migrations/_archived/pre_1.0/mig061/. The consolidated baseline creates the table exactly once (001_advisoryai_consolidated_baseline.sql:736), and the live and fresh shapes of search_feedback are identical in the step 9 column probe.
Known shape divergences — read before converging
These were measured by converging a throwaway copy of the successor baseline into stellaops_advisoryai on 2026-08-28, diffing it against the live schemas, then dropping it. The target was verified empty again afterwards (relations=0, extensions=plpgsql).
D1 — pg_trgm lands in the advisoryai schema, not public (functional; act on it)
On the live source pg_trgm is installed in public, so the two trigram indexes read gin_trgm_ops unqualified. On a freshly converged target it lands in advisoryai, because the migration runner executes every migration under SET LOCAL search_path TO "advisoryai", public (MigrationRunner.ApplyMigrationAsync) and CREATE EXTENSION without a SCHEMA clause takes the first entry of search_path. The resulting indexes read advisoryai.gin_trgm_ops.
Two consequences:
- Production code calls
similarity()unqualified —PostgresKnowledgeSearchStore.cs:192,207(the fuzzy fallback path) andSearchAnalyticsService.cs:353,358. On the target it resolves only because the DSN’sUsername=advisoryaimakes"$user"in the defaultsearch_pathname theadvisoryaischema. Measured: as roleadvisoryai,SELECT similarity('kb chunk','kb chunks')returns0.72727275; underSET search_path TO publicthe same call fails withERROR: function similarity(unknown, unknown) does not exist. On the source it succeeds under either path. Anything that stops the effective search_path from includingadvisoryai— a different login role, an explicitSearch Path=in the DSN, a maintenance session — breaks trigram search on the target and not on the source. DROP SCHEMA advisoryai CASCADEdropspg_trgmwith it. Measured during teardown:NOTICE: drop cascades to 28 other objects … drop cascades to extension pg_trgm.
Step 3 pins the extension into public before convergence so the target reproduces the source placement; step 9 asserts it.
D2 — column ordinal order differs on kb_chunk and search_events (survivable; assert it)
The column set, type, nullability and default are identical; only ordinal positions differ, because the live schema grew by ALTER TABLE … ADD COLUMN while the fresh baseline declares the final order. Live kb_chunk carries embedding, metadata, indexed_at at 11-13 and the five per-language tsvectors at 18-22; the fresh baseline carries the tsvectors at 11-15 and embedding, metadata … indexed_at at 16-22. search_events differs the same way on created_at.
This is survivable only because pg_dump always emits an explicit column list. Measured on this server (PostgreSQL 18.1): pg_dump --data-only produced COPY advisoryai._aai9_probe (a, b, c) FROM stdin; and --column-inserts produced INSERT INTO advisoryai._aai9_probe (a, b, c) VALUES (…). Positional COPY/INSERT, a hand-written \copy without a column list, and any CSV round-trip that reorders columns are forbidden in this window.
D3 — ledger-only difference (no action; not copied)
The target’s advisoryai.schema_migrations has a normalized_checksum column the live one lacks (the current MigrationChecksumReconciler.LedgerDdl adds it). The ledger is never copied.
D4 — expected, intended differences
- Live-only tables
advisoryai.__migration_historyandopsmemory.schema_migrations— excluded. - Target-only tables
advisoryai.evidence_packsandadvisoryai.evidence_pack_run_links— converge empty. - The target carries 25 P13
retention-classtable comments; the live source carries 1. That is the baseline doing what P13 requires, not drift. - Object ownership: every live object is owned by the
stellaopssuperuser; every target object is owned byadvisoryai. That inversion is the point of the move.
D5 — pgvector: nothing to preserve, nothing to degrade
pg_available_extensions has no vector/pgvector/vectors row on this PostgreSQL 18.1 server, so the extension cannot be installed anywhere in this estate. The baseline’s conditional block (001_advisoryai_consolidated_baseline.sql:205-226) is therefore a measured no-op on both sides: kb_chunk.embedding_vec exists in neither database, there is no HNSW index, and both advisoryai.kb_chunk.embedding and opsmemory.decisions.similarity_vector are real[] (_float4). PostgresKnowledgeSearchStore and UnifiedSearchIndexer both probe information_schema for embedding_vec and take the array path (PostgresKnowledgeSearchStore.cs:609, UnifiedSearchIndexer.cs:471). No column in the copy set can silently degrade to a different type, and kb_chunk carries 13 indexes on both sides.
D6 — the role-identity trap: converge and verify under the SERVICE role, never the superuser
Precedent: the NTF-9 live window, 2026-08-28. notify-web was cut over to its own database as its own least-privilege role and crash-looped. StartupMigrationHost.ApplyMigrationAsync runs migration DML; the notify schema has FORCE ROW LEVEL SECURITY on 21 tables whose policies call notify_app.require_current_tenant(); and FORCE RLS subjects even the table owner to the policy, so the migration raised P0001: app.tenant_id session variable not set and the host died. It had never been seen because notify-web had always connected as the cluster superuser, and superusers bypass RLS unconditionally. The window was rolled back.
AdvisoryAI is clear on this axis — measured, not inherited. Re-run the survey before the window:
for schema in advisoryai opsmemory; do
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -U "${AAI9_PG_OPERATOR}" -d "${AAI9_SOURCE_DB}" -c \
"SELECT '${schema} '
|| count(*) FILTER (WHERE relforcerowsecurity) || ' forced / '
|| count(*) FILTER (WHERE relrowsecurity) || ' enabled of '
|| count(*)
FROM pg_class WHERE relnamespace = '${schema}'::regnamespace AND relkind = 'r';"
done
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -U "${AAI9_PG_OPERATOR}" -d "${AAI9_SOURCE_DB}" -c \
"SELECT count(*) FROM pg_policy p
JOIN pg_class c ON c.oid = p.polrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname IN ('advisoryai','opsmemory');"
Verified by: advisoryai 0 forced / 0 enabled of 26, opsmemory 0 forced / 0 enabled of 2, and zero pg_policy rows (measured 2026-08-28). Any non-zero result means this window has acquired the NTF-9 failure mode and the converge boot must be re-planned before it is attempted. For calibration, the same query on the schemas that will hit this wall returned notify 21 forced / 21 enabled of 41, scheduler 13/13 of 21, and binaries 11 forced / 14 enabled of 16.
The discipline still applies even though AdvisoryAI is clear, because a clean survey today is not a converge-time guarantee: converge and verify the target under the advisoryai role, never under stellaops. In the real window the service role is what advisoryai-web connects as, so the converge boot has this property by construction — the rule matters for every manual psql check an operator runs alongside it.
Why an un-roled superuser session is a false green, and what is sufficient instead — measured in the sandbox with a FORCE ROW LEVEL SECURITY table owned by advisoryai whose policy raises P0001 when app.tenant_id is unset:
| Session | Result |
|---|---|
stellaops superuser, no SET ROLE | INSERT SUCCEEDED — the failure is invisible |
stellaops superuser + SET ROLE advisoryai | ERROR: app.tenant_id session variable not set |
real advisoryai login (session_user=advisoryai, is_superuser=off) | ERROR: app.tenant_id session variable not set — identical |
So SET ROLE advisoryai reproduces the defect exactly as a real service-role login does; only the un-roled superuser session hides it. Either is acceptable for a manual check, and the first row is the positive control proving the check can fail.
The container’s pg_hba.conf has host all all 127.0.0.1/32 trust, so an operator can take a real service-role session without handling the password at all:
docker exec "${AAI9_PG_CONTAINER}" psql -h 127.0.0.1 -U "${AAI9_TARGET_ROLE}" -d "${AAI9_TARGET_DB}" \
-tAc "SELECT current_user, session_user, current_setting('is_superuser');"
Verified by: it prints advisoryai|advisoryai|off. Use that session for every target-side check in steps 3 and 9. Note the deliberate exception: the restore in step 8 runs as the operator role, because pg_restore --no-owner writes into tables owned by advisoryai. That is safe here precisely because the survey above returned zero forced-RLS tables; on a FORCE-RLS schema it would be the NTF-9 bug in reverse.
Required approvals and real receipts
The window is NO-GO until every row names a real reviewed artifact. Blank, TBD, verbal, or source-only evidence is a failure.
| Required input | Receipt |
|---|---|
| Window start/end, owner, DB operator, rollback owner | approved change/window ID |
| Reviewed successor image builds | immutable @sha256: digests for advisoryai-web and advisoryai-worker, built from an isolated git worktree at HEAD |
| Source composition | commit carrying AAI-9 gates 1-4 and 9-11; targeted AdvisoryAI-family conformance green |
| Database provision | stellaops_advisoryai + role advisoryai, NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS, sibling CONNECT revocations — satisfied 2026-08-28; re-assert in step 3 |
| Backup | full source dump SHA-256 plus an isolated restore-test receipt on the same PostgreSQL major (18) |
| Rollback | the three predecessor image IDs, exact Compose config_files chains, environment/config blob digests, forward and rollback grant plans |
No route swap and no claimant coordination. Both were retired by RAR-7 8bebf33263; the swap script exits 2 on every path. Step 11 confirms the auto-published surface instead. The manifest is still read for its surviving route-table findings — see its retirement block. | |
| Activation dependencies | Doctor grant/capability-row plan, Console prefix rebuild plan, evidence-pack export through the real Router transport (F-AAI5-6) |
1. Establish the workspace
Use opaque non-secret identifiers.
set -euo pipefail
export AAI9_WINDOW_ID='<approved-window-id>'
export AAI9_ROOT="tmp/aai9-${AAI9_WINDOW_ID}"
export AAI9_PG_CONTAINER='stellaops-postgres'
export AAI9_PG_OPERATOR='stellaops'
export AAI9_SOURCE_DB='stellaops_platform'
export AAI9_TARGET_DB='stellaops_advisoryai'
export AAI9_TARGET_ROLE='advisoryai'
mkdir -p "${AAI9_ROOT}"/{preflight,backup,parity,rollback,activation}
test -n "${AAI9_WINDOW_ID}"
git rev-parse HEAD | tee "${AAI9_ROOT}/preflight/repository-head.txt"
git status --short | tee "${AAI9_ROOT}/preflight/repository-status.txt"
Verified by: repository-head.txt is a 40-character SHA and every row of repository-status.txt is explained in the window record. Never clean or reset the checkout — this is a shared clone.
Attach the approvals under preflight/ and prove they are non-empty:
# `route-claimant-coordination.txt` was REMOVED from this list on 2026-09-04 (F-AAI9-15): it
# demanded the F-AAI5-5 `/api/v1/advisories` coordination receipt, a precondition RAR-7
# (`8bebf33263`, 2026-09-02) retired. Leaving it here made the window unstartable on a receipt
# nobody can produce.
for receipt in \
window-approval.txt backup-restore-proof.txt grants-forward.sql grants-rollback.sql \
advisoryai-web-image.txt advisoryai-worker-image.txt; do
test -s "${AAI9_ROOT}/preflight/${receipt}" || { echo "NO-GO: missing ${receipt}" >&2; exit 1; }
done
grep -Eq '@sha256:[0-9a-f]{64}$' "${AAI9_ROOT}/preflight/advisoryai-web-image.txt"
grep -Eq '@sha256:[0-9a-f]{64}$' "${AAI9_ROOT}/preflight/advisoryai-worker-image.txt"
Verified by: the loop exits 0 and both grep -Eq succeed. A floating tag in either image receipt is NO-GO (§1.5: deploy by digest, never by tag).
2. Capture rollback topology before changing anything
All three predecessor containers must exist and be running. The successor containers must be absent or stopped. Record the exact image IDs and Compose chains; do not reconstruct a rollback from a sibling service.
for container in stellaops-advisory-ai-web stellaops-advisory-ai-worker stellaops-opsmemory-web; do
docker inspect "${container}" --format '{{.Config.Image}} {{.Image}}' \
> "${AAI9_ROOT}/rollback/${container}.image.txt"
docker inspect "${container}" \
--format '{{index .Config.Labels "com.docker.compose.project.config_files"}}' \
> "${AAI9_ROOT}/rollback/${container}.config-files.txt"
docker inspect "${container}" --format '{{json .Config.Env}}' \
| sha256sum > "${AAI9_ROOT}/rollback/${container}.environment.sha256"
docker inspect "${container}" --format '{{json .Mounts}}' \
> "${AAI9_ROOT}/rollback/${container}.mounts.json"
docker inspect "${container}" --format '{{json .NetworkSettings.Networks}}' \
> "${AAI9_ROOT}/rollback/${container}.networks.json"
test -s "${AAI9_ROOT}/rollback/${container}.config-files.txt"
done
docker ps --format '{{.Names}}\t{{.Image}}\t{{.Status}}' | sort \
| tee "${AAI9_ROOT}/preflight/containers-before.tsv"
The environment receipt is a hash only, because the raw environment carries the predecessor’s ConnectionStrings__Default and Authority secrets. The protected configuration source stays in its approved secret store.
Verified by: three non-empty .image.txt files, three non-empty .config-files.txt files, and containers-before.tsv listing all three predecessors as Up.
Measure the three carried volumes now — AAI-8 recorded them at 0 files / 0 bytes on 2026-08-22, and that is a boundary measurement, not a standing fact:
for volume in advisory-ai-queue advisory-ai-plans advisory-ai-outputs; do
docker run --rm -v "${volume}:/v:ro" busybox sh -c \
'printf "%s files %s bytes\n" "$(find /v -type f | wc -l)" "$(du -sb /v | cut -f1)"' \
| sed "s|^|${volume} |"
done | tee "${AAI9_ROOT}/preflight/carried-volumes.txt"
Verified by: carried-volumes.txt has three rows. A non-zero count is not a failure; it is queued work that the swap must carry, and it must be re-measured after the repoint in step 10.
3. Converge an empty target from the successor’s own migrations
The target must converge from StellaOps.AdvisoryAI.Persistence, never by copying the source schema. Assert the boundary is still intact and the target is still empty before anything else.
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_TARGET_DB}" -c \
"SELECT count(*) FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname NOT IN ('pg_catalog','information_schema')
AND n.nspname NOT LIKE 'pg_toast%';" \
| tee "${AAI9_ROOT}/preflight/target-relations-before.txt"
test "$(cat "${AAI9_ROOT}/preflight/target-relations-before.txt")" = 0
bash tools/scripts/deploy/postgres/probe-database-isolation.sh \
| tee "${AAI9_ROOT}/preflight/database-isolation.txt"
grep -q 'PASS' "${AAI9_ROOT}/preflight/database-isolation.txt"
Verified by: target-relations-before.txt is exactly 0 and the isolation probe reports PASS. A non-zero relation count means someone converged or seeded the target; stop and reconcile — do not restore into it.
Pin pg_trgm into public first (finding D1). Without this the extension lands in the advisoryai schema and the target diverges from the source in a way that survives every row-parity check:
Run it as the service role (finding D6), not as the operator. The role can do this: pg_trgm is a trusted extension (trusted = true in its control file) and advisoryai owns the database, so it owns public through pg_database_owner. Measured — the statement succeeds and the result is pg_trgm in public owned by advisoryai. No operator escalation is needed for this step.
docker exec "${AAI9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
-h 127.0.0.1 -U "${AAI9_TARGET_ROLE}" -d "${AAI9_TARGET_DB}" \
-c 'CREATE EXTENSION IF NOT EXISTS pg_trgm SCHEMA public;' \
| tee "${AAI9_ROOT}/preflight/target-pg-trgm.txt"
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
-h 127.0.0.1 -U "${AAI9_TARGET_ROLE}" -d "${AAI9_TARGET_DB}" -c \
"SELECT n.nspname FROM pg_extension e JOIN pg_namespace n ON n.oid = e.extnamespace
WHERE e.extname = 'pg_trgm';"
Verified by: the last query prints public. CREATE EXTENSION IF NOT EXISTS inside the baseline then becomes a no-op — the converge log shows NOTICE: extension "pg_trgm" already exists, skipping — and the baseline’s pg_extension-derived schema lookup (001_advisoryai_consolidated_baseline.sql:182-201) builds both trigram indexes against public.gin_trgm_ops. Measured effect of this pin: with it, the index comparison in step 9e is 85/85 with an empty diff; without it, the same comparison carries the two D1 divergences.
Also expect NOTICE: pgvector is unavailable; AdvisoryAI Knowledge Search uses array embeddings. in the converge log — the baseline reporting D5 from inside the migration. Its absence would mean the extension became available and D5 needs re-deciding.
Now converge by booting only advisoryai-web on the reviewed digest-pinned overlay, with knowledge auto-index disabled. The overlay fails closed without STELLAOPS_POSTGRES_ADVISORYAI_CONNECTION (devops/compose/docker-compose.advisoryai.yml:86), which is what makes it impossible for this boot to converge anything but its own database.
The four carried-table writers are now default-off in the overlay itself (F-AAI9-11, closed 2026-08-28) — the x-advisoryai-carried-table-writers anchor. Nothing to add to the environment for the converge boot; the operator’s job here is to assert it, because a default that holds only because nobody set it is not a gate:
Key (section AdvisoryAI:KnowledgeSearch, prefix ADVISORYAI__) | Default in code | What it writes if left on |
|---|---|---|
KnowledgeAutoIndexOnStartup | true | INSERTs kb_doc, kb_chunk, api_spec, api_operation, doctor_search_projection |
SearchQualityMonitorEnabled | true | INSERT/UPDATEs search_quality_alerts (SearchQualityMonitor.cs:1182,1199) |
SearchAnalyticsRetentionEnabled | true | DELETEs from search_events, search_history, search_feedback, search_quality_alerts on a 90-day cutoff |
UnifiedAutoIndexOnStartup | true | inert only while no adapter base URL is set (UnifiedSearchIndexRefreshService.cs:27-38); one URL re-arms it |
Precision note on the fourth key, added 2026-09-04 — the render check does not fully cover it. UnifiedSearchIndexRefreshService returns early only when UnifiedAutoIndexEnabled is false and no FindingsAdapterBaseUrl/VexAdapterBaseUrl/PolicyAdapterBaseUrl is set (:27-33). Past that gate, UnifiedAutoIndexOnStartup suppresses the startup rebuild (:45) but not the periodic SafeIndexAsync loop that follows it (:51-57). So the key the overlay pins false and StagedComposeRender_DefaultsEveryCarriedTableWriterOff asserts is not what keeps this writer inert during the converge boot — the absent adapter URLs and the false default of the separate UnifiedAutoIndexEnabled are. Today that holds: the overlay sets no adapter base URL (grep -n 'AdapterBaseUrl' devops/compose/docker-compose.advisoryai.yml → no match) and KnowledgeSearchOptions.UnifiedAutoIndexEnabled defaults false (:81). Assert it rather than inherit it: if the converge-boot render carries any adapter base URL, or ADVISORYAI__AdvisoryAI__KnowledgeSearch__UnifiedAutoIndexEnabled is not absent-or-false, this writer will run on a timer during the boot and the guard test will still be green. The step 3 tables_checked=27 rows=0 assertion is the backstop that catches it either way.
All four live in the web host only — the worker calls AddAdvisoryAiCore but neither AddUnifiedSearch nor KnowledgeSearchStartupRebuildService, so it hosts none of them. StagedComposeRender_DefaultsEveryCarriedTableWriterOff pins both halves (present-and-false on the web key, absent on the worker key).
SearchAnalyticsRetentionEnabled deserves separate attention: it is the only one that is also a post-restore hazard. Left on after step 8 it would silently prune migrated rows older than the cutoff from four carried tables. All four are empty today, so today’s blast radius is nil — if the step-3 re-measurement finds rows in any of them, treat re-enabling retention in step 10 as a decision, not a default.
Render, assert, boot, wait, capture, stop:
test -s "${AAI9_ROOT}/preflight/advisoryai-compose-render.json"
jq -e '
.services["advisoryai-web"].environment as $web |
$web.STELLAOPS_POSTGRES_ADVISORYAI_CONNECTION != null and
([ "KnowledgeAutoIndexOnStartup",
"UnifiedAutoIndexOnStartup",
"SearchQualityMonitorEnabled",
"SearchAnalyticsRetentionEnabled" ]
| all($web["ADVISORYAI__AdvisoryAI__KnowledgeSearch__" + .] == "false")) and
($web["Doctor__Registration__Enabled"] // "false") == "false" and
($web | has("ConnectionStrings__Default") | not)
' "${AAI9_ROOT}/preflight/advisoryai-compose-render.json"
# Execute with the reviewed Compose array from advisoryai-deploy-aai8-staged-stack.md,
# never a reconstructed shorthand. The legacy hosts are still running at this point, so the
# overlay must be composed WITHOUT docker-compose.stella-services.yml (alias collision).
docker compose '<recorded -p/--project-directory/--env-file/-f chain>' up -d --no-deps advisoryai-web
until test "$(docker inspect -f '{{.State.Health.Status}}' stellaops-advisoryai-web)" = healthy; do sleep 2; done
docker logs stellaops-advisoryai-web > "${AAI9_ROOT}/preflight/advisoryai-web-converge.log" 2>&1
docker stop stellaops-advisoryai-web | tee "${AAI9_ROOT}/preflight/converge-stop.txt"
Prove the convergence did what it claims, and that it wrote no data:
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -F '|' -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_TARGET_DB}" -c \
"SELECT migration_name, category, checksum FROM advisoryai.schema_migrations ORDER BY 1;" \
| tee "${AAI9_ROOT}/preflight/target-ledger.txt"
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_TARGET_DB}" -c \
"SELECT to_regclass('advisoryai.__migration_history') IS NULL
AND to_regclass('opsmemory.schema_migrations') IS NULL;"
Verified by: target-ledger.txt has exactly two rows — 001_advisoryai_consolidated_baseline.sql and 002_advisoryai_evidence_pack_store.sql, both startup — and the second query prints t. The checksums must equal the SHA-256 of the LF-normalized migration files (MigrationChecksum.Compute); today those are 281a9ab286096f0a5d258c1aba4fe7335bcd4d66ff9f133b026c2062d89bb3dd and 6bff60c2533233a946f89acefb718ef60eeee70ab1e3561816bbdbde489935eb, which are also the SHA-256s of the two embedded resources inside the built StellaOps.AdvisoryAI.Persistence.dll. A third ledger row, or a __migration_history table, means a legacy migration authority reached the target: stop and diagnose before copying.
PRECONDITION FOR STEP 8 — the converged target must hold ZERO rows outside its ledger. This is the assertion that actually closes F-AAI9-11: the render check above proves the configuration, and this proves the outcome. Run it after the converge boot has been stopped, and do not proceed to the dump until it passes.
AAI9_TARGET_COUNT_SQL="$(docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
-h 127.0.0.1 -U "${AAI9_TARGET_ROLE}" -d "${AAI9_TARGET_DB}" -c \
\"SELECT string_agg(format('SELECT %L AS t, count(*) AS n FROM %I.%I',
n.nspname||'.'||c.relname, n.nspname, c.relname), ' UNION ALL ')
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname IN ('advisoryai','opsmemory')
AND c.relkind = 'r'
AND c.relname <> 'schema_migrations';\")"
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -F '|' -v ON_ERROR_STOP=1 \
-h 127.0.0.1 -U "${AAI9_TARGET_ROLE}" -d "${AAI9_TARGET_DB}" \
-c "SELECT t, n FROM (${AAI9_TARGET_COUNT_SQL}) x WHERE n <> 0 ORDER BY t;" \
| tee "${AAI9_ROOT}/preflight/target-nonempty-after-converge.txt"
test ! -s "${AAI9_ROOT}/preflight/target-nonempty-after-converge.txt"
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
-h 127.0.0.1 -U "${AAI9_TARGET_ROLE}" -d "${AAI9_TARGET_DB}" \
-c "SELECT 'tables_checked='||count(*)||' rows='||coalesce(sum(n),0)
FROM (${AAI9_TARGET_COUNT_SQL}) x;" \
| tee "${AAI9_ROOT}/preflight/target-empty-proof.txt"
Verified by: target-nonempty-after-converge.txt is empty, and target-empty-proof.txt reads tables_checked=27 rows=0 — the 25 carried tables plus the two target-native Evidence Pack tables, ledger excluded. Measured 2026-08-28 on a rehearsal convergence run under the advisoryai role: exactly that result. A non-zero row anywhere means a writer ran despite the render check; stop, do not restore, and re-derive F-AAI9-11’s instance list — the guard test asserts four writers, and a fifth would look exactly like this.
Re-measure the source now, at the window boundary, and record the receipt the copy contract is checked against:
AAI9_COUNT_SQL="$(docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_SOURCE_DB}" -c \
\"SELECT string_agg(format('SELECT %L AS t, count(*) AS n FROM %I.%I',
n.nspname||'.'||c.relname, n.nspname, c.relname), ' UNION ALL ')
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname IN ('advisoryai','opsmemory') AND c.relkind = 'r';\")"
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -F '|' -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_SOURCE_DB}" \
-c "SELECT t, n FROM (${AAI9_COUNT_SQL}) x ORDER BY t;" \
| tee "${AAI9_ROOT}/parity/source-counts-window-open.txt"
Verified by: the five non-empty rows equal the copy contract (api_operation|2190, api_spec|1, doctor_search_projection|186, kb_chunk|10154, kb_doc|707) and every other carried table is 0. Any drift is fine — record the new numbers and use them for the rest of the window — but an unrecorded drift is a failure of this step.
4. Capture a full recovery backup
The parity dump in step 7 is a transfer artifact, not a backup. Take a real one first.
docker exec "${AAI9_PG_CONTAINER}" pg_dump \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_SOURCE_DB}" -Fc \
--schema=advisoryai --schema=opsmemory \
> "${AAI9_ROOT}/backup/source-advisoryai-full.dump"
test -s "${AAI9_ROOT}/backup/source-advisoryai-full.dump"
sha256sum "${AAI9_ROOT}/backup/source-advisoryai-full.dump" \
| tee "${AAI9_ROOT}/backup/source-advisoryai-full.dump.sha256"
pg_restore --list "${AAI9_ROOT}/backup/source-advisoryai-full.dump" \
> "${AAI9_ROOT}/backup/source-advisoryai-full.list"
grep -c 'TABLE DATA' "${AAI9_ROOT}/backup/source-advisoryai-full.list"
Verified by: a non-empty dump, a recorded SHA-256, and an isolated restore test on a scratch PostgreSQL 18 database that reproduces the 13,238-row carried total. A dump nobody has restored is not a backup; attach the restore transcript as preflight/backup-restore-proof.txt.
CREATE
pg_trgmINpublicON THE RESTORE TARGET FIRST, OR THIS BACKUP DOES NOT RESTORE (measured 2026-09-04, window20260904-aai9— the restore test failed on its first attempt and this is the reason). The dump hard-codes the source-side extension placement (finding D1), so a fresh database without it aborts:pg_restore: error: could not execute query: ERROR: operator class "public.gin_trgm_ops" does not exist for access method "gin" Command was: CREATE INDEX idx_kb_chunk_body_trgm ON advisoryai.kb_chunk USING gin (body public.gin_trgm_ops);So the restore test — and any real recovery — runs
CREATE EXTENSION IF NOT EXISTS pg_trgm SCHEMA public;against the target beforepg_restore. This is the documented recovery path, so it failing at the moment it is needed is the whole cost of leaving it unwritten. Measured second attempt: exit 0, zero error lines, 7.06 s,tables_checked=25 rows=13238.
5. Fence the three predecessor writers and freeze the source
Stop, but do not remove, all three predecessor containers. The successor containers stay stopped.
docker stop stellaops-advisory-ai-web stellaops-advisory-ai-worker stellaops-opsmemory-web \
| tee "${AAI9_ROOT}/preflight/writer-fence-stop.txt"
for container in stellaops-advisory-ai-web stellaops-advisory-ai-worker stellaops-opsmemory-web \
stellaops-advisoryai-web stellaops-advisoryai-worker; do
if docker inspect "${container}" >/dev/null 2>&1; then
test "$(docker inspect -f '{{.State.Running}}' "${container}")" = false
fi
done
docker exec "${AAI9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_SOURCE_DB}" -c \
"SELECT pid, usename, application_name, client_addr, state, query_start
FROM pg_stat_activity
WHERE datname = '${AAI9_SOURCE_DB}'
AND application_name ILIKE '%advisory%'
ORDER BY pid;" \
| tee "${AAI9_ROOT}/preflight/source-sessions-after-fence.txt"
Verified by: the docker stop output names all three containers, the test loop exits 0 for every container that exists, and source-sessions-after-fence.txt lists no AdvisoryAI session. stellaops_platform serves many other services — do not expect an empty pg_stat_activity, only an AdvisoryAI-free one.
Build canonical fingerprints for the 25 carried tables and capture them twice, 30 seconds apart:
aai9_fingerprints() {
local database="$1" output="$2" schema table qualified count digest
: > "${output}"
for qualified in "${AAI9_COPY_ADVISORYAI[@]/#/advisoryai.}" \
"${AAI9_COPY_OPSMEMORY[@]/#/opsmemory.}"; do
schema="${qualified%%.*}"; table="${qualified##*.}"
count="$(docker exec "${AAI9_PG_CONTAINER}" psql -XqAt -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${database}" \
-c "SELECT count(*) FROM ${schema}.${table};")"
digest="$(docker exec "${AAI9_PG_CONTAINER}" psql -XqAt -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${database}" \
-c "COPY (SELECT to_jsonb(t)::text FROM ${schema}.${table} AS t
ORDER BY to_jsonb(t)::text COLLATE \"C\") TO STDOUT" \
| sha256sum | awk '{print $1}')"
printf '%s\t%s\t%s\n' "${qualified}" "${count}" "${digest}" >> "${output}"
done
}
aai9_fingerprints "${AAI9_SOURCE_DB}" "${AAI9_ROOT}/parity/source-a.tsv"
sleep 30
aai9_fingerprints "${AAI9_SOURCE_DB}" "${AAI9_ROOT}/parity/source-b.tsv"
cmp "${AAI9_ROOT}/parity/source-a.tsv" "${AAI9_ROOT}/parity/source-b.tsv"
Verified by: cmp exits 0 and source-a.tsv has 25 rows. to_jsonb renders every column by name, so the digest is immune to the D2 ordinal divergence — that is why it is the right comparator here. Any difference between the two samples means the writer fence is incomplete: do not copy.
6. Pin sequence and identity ownership before the dump
Measured 2026-08-28: both sides carry zero sequences and zero identity/serial columns. That removes the entire setval class of risk from this window — but the check must still run, so a future migration that adds one fails safely instead of silently.
aai9_sequence_inventory() {
docker exec "${AAI9_PG_CONTAINER}" psql -XqAt -F $'\t' -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "$1" -c \
"SELECT format('%I.%I', sn.nspname, s.relname),
format('%I.%I', tn.nspname, t.relname),
a.attname,
CASE a.attidentity WHEN 'a' THEN 'identity-always'
WHEN 'd' THEN 'identity-default'
ELSE 'owned-sequence' END
FROM pg_class s
JOIN pg_namespace sn ON sn.oid = s.relnamespace
JOIN pg_depend d ON d.classid = 'pg_class'::regclass AND d.objid = s.oid
AND d.refclassid = 'pg_class'::regclass AND d.deptype IN ('a','i')
JOIN pg_class t ON t.oid = d.refobjid
JOIN pg_namespace tn ON tn.oid = t.relnamespace
JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = d.refobjsubid
WHERE s.relkind = 'S' AND sn.nspname IN ('advisoryai','opsmemory')
ORDER BY 1,2,3;" > "$2"
}
aai9_sequence_inventory "${AAI9_SOURCE_DB}" "${AAI9_ROOT}/parity/source-sequences.tsv"
aai9_sequence_inventory "${AAI9_TARGET_DB}" "${AAI9_ROOT}/parity/target-sequences.before.tsv"
cmp "${AAI9_ROOT}/parity/source-sequences.tsv" "${AAI9_ROOT}/parity/target-sequences.before.tsv"
Verified by: cmp exits 0. Today both files are empty, which is the expected result. A non-empty inventory means a sequence appeared since this runbook was written: add its --table= argument to the dump in step 7 and compare last_value/is_called in step 9 before accepting parity. A missing, extra, renamed, or differently owned sequence is NO-GO.
7. Create the fenced data-only dump
AAI9_DUMP_ARGS=()
for table in "${AAI9_COPY_ADVISORYAI[@]}"; do AAI9_DUMP_ARGS+=(--table="advisoryai.${table}"); done
for table in "${AAI9_COPY_OPSMEMORY[@]}"; do AAI9_DUMP_ARGS+=(--table="opsmemory.${table}"); done
while IFS=$'\t' read -r sequence _; do
test -n "${sequence}" && AAI9_DUMP_ARGS+=(--table="${sequence}")
done < "${AAI9_ROOT}/parity/source-sequences.tsv"
test "${#AAI9_DUMP_ARGS[@]}" -ge 25
docker exec "${AAI9_PG_CONTAINER}" pg_dump \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_SOURCE_DB}" -Fc \
--data-only --no-owner --no-privileges \
--exclude-table=advisoryai.schema_migrations \
--exclude-table=advisoryai.__migration_history \
--exclude-table=opsmemory.schema_migrations \
"${AAI9_DUMP_ARGS[@]}" \
> "${AAI9_ROOT}/backup/source-advisoryai-fenced-data.dump"
test -s "${AAI9_ROOT}/backup/source-advisoryai-fenced-data.dump"
sha256sum "${AAI9_ROOT}/backup/source-advisoryai-fenced-data.dump" \
| tee "${AAI9_ROOT}/backup/source-advisoryai-fenced-data.dump.sha256"
pg_restore --list "${AAI9_ROOT}/backup/source-advisoryai-fenced-data.dump" \
> "${AAI9_ROOT}/backup/source-advisoryai-fenced-data.list"
! grep -Eq 'TABLE DATA (advisoryai|opsmemory) (schema_migrations|__migration_history)' \
"${AAI9_ROOT}/backup/source-advisoryai-fenced-data.list"
! grep -Eq 'TABLE DATA advisoryai evidence_pack' \
"${AAI9_ROOT}/backup/source-advisoryai-fenced-data.list"
Verified by: the dump is non-empty, its SHA-256 is recorded, and both ! grep -Eq assertions hold — no ledger table and no Evidence Pack table is in the archive. The --exclude-table flags are a safety belt on top of the explicit --table list and an auditable statement of intent; both must be present.
Do not add --column-inserts, --inserts, or any hand-written \copy. The custom-format archive replays COPY … (col, col, …) with an explicit column list, which is what makes the D2 ordinal divergence harmless.
8. Restore once, atomically, into the empty target
All five containers remain stopped. Do not truncate or merge a non-empty target.
for container in stellaops-advisory-ai-web stellaops-advisory-ai-worker stellaops-opsmemory-web \
stellaops-advisoryai-web stellaops-advisoryai-worker; do
if docker inspect "${container}" >/dev/null 2>&1; then
test "$(docker inspect -f '{{.State.Running}}' "${container}")" = false
fi
done
docker exec -i "${AAI9_PG_CONTAINER}" pg_restore \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_TARGET_DB}" \
--single-transaction --exit-on-error --data-only --no-owner --no-privileges \
< "${AAI9_ROOT}/backup/source-advisoryai-fenced-data.dump" \
2>&1 | tee "${AAI9_ROOT}/backup/target-restore.txt"
Verified by: pg_restore exits 0 and target-restore.txt contains no error. Do not retry over a partially changed target: --single-transaction --exit-on-error is precisely the property that makes one retry against the still-empty target safe after the failure is diagnosed.
The restore connects as the operator role while every target object is owned by advisoryai (finding D4). That is intended — --no-owner --no-privileges keeps the converged ownership — but it means the operator role must retain superuser or explicit INSERT on the target for the duration of this step.
9. Prove exhaustive parity before any source retirement
9a. The source is unchanged
aai9_fingerprints "${AAI9_SOURCE_DB}" "${AAI9_ROOT}/parity/source-after.tsv"
cmp "${AAI9_ROOT}/parity/source-a.tsv" "${AAI9_ROOT}/parity/source-after.tsv"
Verified by: cmp exits 0. A difference means a writer escaped the fence during the copy.
9b. Row counts and content digests match on all 25 tables
aai9_fingerprints "${AAI9_TARGET_DB}" "${AAI9_ROOT}/parity/target-after.tsv"
diff -u "${AAI9_ROOT}/parity/source-after.tsv" "${AAI9_ROOT}/parity/target-after.tsv" \
| tee "${AAI9_ROOT}/parity/table-parity.diff"
test ! -s "${AAI9_ROOT}/parity/table-parity.diff"
grep -P '^advisoryai\.kb_chunk\t10154\t' "${AAI9_ROOT}/parity/target-after.tsv"
awk -F'\t' '{s+=$2} END {print s}' "${AAI9_ROOT}/parity/target-after.tsv"
Split the same fingerprint file into the two classes the copy contract names, and record BOTH subtotals — this is the number the window closeout carries forward:
grep -F -f <(printf '%s\n' "${AAI9_DERIVED[@]}") "${AAI9_ROOT}/parity/target-after.tsv" \
> "${AAI9_ROOT}/parity/derived-at-fence.tsv"
grep -F -v -f <(printf '%s\n' "${AAI9_DERIVED[@]}") "${AAI9_ROOT}/parity/target-after.tsv" \
> "${AAI9_ROOT}/parity/invariant-at-fence.tsv"
test "$(wc -l < "${AAI9_ROOT}/parity/derived-at-fence.tsv")" -eq 5
test "$(wc -l < "${AAI9_ROOT}/parity/invariant-at-fence.tsv")" -eq 20
echo "derived-at-fence: $(awk -F'\t' '{s+=$2} END {print s+0}' "${AAI9_ROOT}/parity/derived-at-fence.tsv")"
echo "invariant-at-fence: $(awk -F'\t' '{s+=$2} END {print s+0}' "${AAI9_ROOT}/parity/invariant-at-fence.tsv")"
Verified by: table-parity.diff is empty, the kb_chunk row is present with its count, and the sum equals the carried total recorded at window open (13,238 as of 2026-08-28). kb_chunk is the only table on this programme carrying real volume, so its digest is called out explicitly rather than trusted to the aggregate. The two subtotals must add to that total and the two files must hold 5 and 20 rows; a different split means AAI9_DERIVED has drifted from the writer’s actual target list and step 10d will assert the wrong thing.
Say this in the same sentence as the total, every time it is reported (AAI-12): the 13,238 is a copy-time proof. Five of the 25 tables — the five carrying all the volume — are a derived index that step 10 rebuilds, so this number stops describing the database a few minutes after step 10c. Step 10d is what proves the move afterwards, and it proves a different, smaller claim: that the 20 INVARIANT tables are still byte-identical. Do not carry 13,238 forward as the post-window result.
9c. Target-only tables are still empty, and no ledger row was copied
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_TARGET_DB}" -c \
"SELECT (SELECT count(*) FROM advisoryai.evidence_packs) = 0
AND (SELECT count(*) FROM advisoryai.evidence_pack_run_links) = 0
AND (SELECT count(*) FROM advisoryai.schema_migrations) = 2
AND to_regclass('advisoryai.__migration_history') IS NULL
AND to_regclass('opsmemory.schema_migrations') IS NULL;"
Verified by: the query prints t.
9d. Extension placement and the trigram operator actually resolve (finding D1)
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_TARGET_DB}" -c \
"SELECT n.nspname = 'public' FROM pg_extension e
JOIN pg_namespace n ON n.oid = e.extnamespace WHERE e.extname = 'pg_trgm';"
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_TARGET_DB}" -c \
"SET search_path TO public; SELECT similarity('kb chunk','kb chunks') > 0;"
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -F '|' -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_TARGET_DB}" -c \
"SELECT indexname, indexdef FROM pg_indexes
WHERE schemaname = 'advisoryai' AND indexname LIKE '%trgm%' ORDER BY 1;"
Verified by: the first query prints t, the second prints t (this is the assertion that fails if step 3’s pin was skipped), and both indexdef rows read gin_trgm_ops unqualified, matching the source. kb_chunk must report 13 indexes on both sides.
9e. Shape parity across every remaining dimension
Run each probe against both databases and diff, excluding the four tables that legitimately differ in membership (advisoryai.__migration_history, opsmemory.schema_migrations, advisoryai.evidence_packs, advisoryai.evidence_pack_run_links) — and advisoryai.schema_migrations, see below.
ADD
advisoryai.schema_migrationsTO THE EXCLUSION LIST, OR THE ORDINAL PROBE FIRES ON ITS OWN DOCUMENTED DIVERGENCE (measured 2026-09-04, window20260904-aai9). Finding D3 guarantees the target ledger carries anormalized_checksumcolumn the live one lacks, so it appears in the columns probe (260 vs 261 rows) and in the ordinal probe — where the rule below reads “a third table in this diff is NO-GO”. Measured membership of the ordinal diff:advisoryai.kb_chunk24 lines,advisoryai.search_events8 lines, andadvisoryai.schema_migrationsone line —+advisoryai.schema_migrations.normalized_checksum|7, the appended column, which shifts no existing ordinal. The ledger is not a carried table and is never copied, so it belongs in the exclusion list for both probes. Left unexcluded, the rule forces an operator to either halt a healthy window or wave the NO-GO through by hand — and the second habit is what the rule exists to prevent.
| Probe | Expected result |
|---|---|
information_schema.columns joined to pg_attribute for format_type, is_nullable, column_default, identity_generation, generation_expression | 260 rows both sides, empty diff except advisoryai.schema_migrations.normalized_checksum on the target (D3) |
pg_indexes.indexdef (whitespace-normalized) | 85 rows both sides, empty diff after step 3’s pg_trgm pin |
pg_get_constraintdef over pg_constraint | 201 rows both sides, empty diff |
pg_get_triggerdef over non-internal pg_trigger | 0 rows both sides |
pg_class.relrowsecurity / relforcerowsecurity / relpersistence | 26 rows both sides, empty diff; 0 forced / 0 enabled on both (finding D6 — re-run the survey here, not only before the window; F-AAI7-3: this baseline ships neither RLS nor roles, tenant isolation is application-enforced) |
information_schema.columns.ordinal_position | expected non-empty diff on advisoryai.kb_chunk and advisoryai.search_events only (D2). A third table in this diff is NO-GO |
Verified by: every row of that table. The ordinal probe is the one place a non-empty diff is the pass condition, and its exact membership is asserted — that is what stops D2 from being used to wave through an unrelated divergence.
Any failure in 9a-9e: keep the freeze, change no route/grant/flag, enter rollback.
10. Repoint the family onto the consolidated overlay
Only now do the successor containers start for real. The legacy hosts stay stopped: the alias collision on advisoryai.stella-ops.local makes running both a NO-GO.
Turn the four carried-table writers back ON. They are default-off in the overlay for the converge boot (F-AAI9-11), and they are product features, not window machinery: leaving them false past the cutover ships a stale knowledge index and no quality or retention loop. This is the step that flips them, and the render below is what proves it happened — a silent omission here is the failure mode the default-off posture trades against.
THIS FLIP REWRITES THE FIVE TABLES STEP 9 JUST PROVED. THE PARITY GATE IS A COPY-TIME PROPERTY, NOT A STEADY-STATE ONE — SAY SO BEFORE ANYONE RE-COUNTS AND CONCLUDES THE MOVE FAILED (measured 2026-09-04, window
20260904-aai9).KnowledgeAutoIndexOnStartupis the writer this runbook itself lists as “INSERTskb_doc,kb_chunk,api_spec,api_operation,doctor_search_projection— the exact five non-empty carried tables”. Turning it on does exactly that on the next boot:AdvisoryAI knowledge startup rebuild completed: documents=639, chunks=9035, api_specs=1, api_operations=2190, doctor_projections=186, duration_ms=17333Re-fingerprinted after activation: 18 of 25 tables still byte-identical to the frozen source, and all five non-empty carried tables rewritten.
Table After step 8 After step 10 kb_chunk10154 9055 digest changed kb_doc707 659 digest changed; the delta is entirely doc_type='markdown'(500 → 452) — the migrated snapshot was indexed from an older repository state, the rebuild indexed the current corpus. Every otherdoc_typeis identicalapi_operation2190 2190 same count, digest changed — re-derived api_spec1 1 same count, digest changed — re-derived doctor_search_projection186 186 same count, digest changed — re-derived This is not data loss and not a failed copy. Those five are a derived index, rebuildable on demand (
KnowledgeSearchEndpoints.cs:102), and the frozen source still holds 10154/707 intact and retained until AAI-10, so rollback is unaffected. But the window’s headline claim — “13,238 rows migrated and verified” — describes a state that lasts from step 8 to step 10c and no longer holds afterwards for the five tables that carry all the volume. Record the parity numbers as a copy-time proof, and expect these five to diverge immediately after activation. An operator who re-counts after step 10 without this note would reasonably conclude the move had failed.Two further deltas are ordinary post-cutover live traffic, not drift:
search_eventsgains rows from search analytics, andopsmemory.decisionsgains the FF-2 write from step 12.
export ADVISORYAI_KNOWLEDGE_AUTOINDEX_ON_STARTUP=true
export ADVISORYAI_UNIFIED_AUTOINDEX_ON_STARTUP=true
export ADVISORYAI_SEARCH_QUALITY_MONITOR_ENABLED=true
# Retention prunes carried tables on a 90-day cutoff. Flip it to true only after step 3's
# re-measurement confirmed those four tables were empty at window open; otherwise treat it as a
# decision and record it.
export ADVISORYAI_SEARCH_ANALYTICS_RETENTION_ENABLED=true
test -s "${AAI9_ROOT}/activation/advisoryai-compose-render.json"
jq -e '
.services["advisoryai-web"].environment as $web |
$web.STELLAOPS_POSTGRES_ADVISORYAI_CONNECTION != null and
.services["advisoryai-worker"].environment.STELLAOPS_POSTGRES_ADVISORYAI_CONNECTION != null and
([ "KnowledgeAutoIndexOnStartup",
"UnifiedAutoIndexOnStartup",
"SearchQualityMonitorEnabled",
"SearchAnalyticsRetentionEnabled" ]
| all($web["ADVISORYAI__AdvisoryAI__KnowledgeSearch__" + .] == "true")) and
([.services["advisoryai-web"].environment, .services["advisoryai-worker"].environment]
| all(has("ConnectionStrings__Default") | not))
' "${AAI9_ROOT}/activation/advisoryai-compose-render.json"
docker compose '<recorded chain>' up -d --no-deps advisoryai-web advisoryai-worker
for container in stellaops-advisoryai-web stellaops-advisoryai-worker; do
until test "$(docker inspect -f '{{.State.Health.Status}}' "${container}")" = healthy; do sleep 2; done
docker inspect "${container}" --format '{{.Config.Image}} {{.Image}}' \
| tee "${AAI9_ROOT}/activation/${container}.image.txt"
done
Prove the repoint reaches exactly one database:
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -F '|' -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_SOURCE_DB}" -c \
"SELECT count(*) FROM pg_stat_activity
WHERE datname = '${AAI9_SOURCE_DB}' AND usename = '${AAI9_TARGET_ROLE}';"
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -F '|' -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_TARGET_DB}" -c \
"SELECT DISTINCT application_name FROM pg_stat_activity
WHERE datname = '${AAI9_TARGET_DB}' ORDER BY 1;" \
| tee "${AAI9_ROOT}/activation/target-sessions.txt"
Verified by: both containers healthy on the recorded digests; the first query prints 0 (zero sibling-database reach, which the CONNECT revocation already guarantees structurally); and target-sessions.txt names the AdvisoryAI application connections. Re-measure the three carried volumes from step 2 and confirm the counts carried across the swap.
10d. Re-verify parity AFTER the writer flip — the step that makes step 9 survive step 10
(10a-10c are the unlabelled activation, recreate and repoint blocks above; this is the closing sub-step of step 10, and the only one that measures.)
This step exists because without it the window’s own procedure destroys its own verification (AAI-12). Step 9 proves parity on all 25 carried tables; the flip above then re-derives five of them, and the executed window (20260904-aai9, 2026-09-04) measured exactly that. An operator who re-counts after step 10 without this step reasonably concludes the move failed; one who does not re-count carries forward a number that no longer describes the database. Both are avoidable, and this is how.
Wait for the startup rebuild to finish before measuring, or the numbers move under the probe:
docker logs stellaops-advisoryai-web --since 15m 2>&1 \
| grep -E 'AdvisoryAI knowledge startup rebuild completed' \
| tee "${AAI9_ROOT}/activation/knowledge-rebuild.txt"
test -s "${AAI9_ROOT}/activation/knowledge-rebuild.txt"
aai9_fingerprints "${AAI9_TARGET_DB}" "${AAI9_ROOT}/parity/target-after-activation.tsv"
(1) The 20 INVARIANT tables must still be byte-identical to the frozen source. This is the assertion that actually protects the move, and it is a hard gate:
grep -F -v -f <(printf '%s\n' "${AAI9_DERIVED[@]}") \
"${AAI9_ROOT}/parity/target-after-activation.tsv" \
> "${AAI9_ROOT}/parity/invariant-after-activation.tsv"
diff -u "${AAI9_ROOT}/parity/invariant-at-fence.tsv" \
"${AAI9_ROOT}/parity/invariant-after-activation.tsv" \
| tee "${AAI9_ROOT}/parity/invariant-activation.diff"
test ! -s "${AAI9_ROOT}/parity/invariant-activation.diff"
Verified by: invariant-activation.diff is EMPTY. A difference here is a real regression — a writer touched a table nothing should re-derive — and is a stop-the-window condition, not a footnote. Two exceptions are expected and are ordinary post-cutover traffic rather than drift, so allow them explicitly and only them: advisoryai.search_events gains rows from search analytics, and opsmemory.decisions gains the step 12 FF-2 write. If the diff shows only those two tables, record the deltas and continue; anything else stops the window.
(2) The five DERIVED tables are recorded, not asserted. Their digests WILL differ; that is the writer doing its job. What is recorded is the resident count, which is the second of the two numbers the closeout carries:
grep -F -f <(printf '%s\n' "${AAI9_DERIVED[@]}") \
"${AAI9_ROOT}/parity/target-after-activation.tsv" \
> "${AAI9_ROOT}/parity/derived-after-activation.tsv"
paste "${AAI9_ROOT}/parity/derived-at-fence.tsv" \
"${AAI9_ROOT}/parity/derived-after-activation.tsv" \
| awk -F'\t' '{printf "%-40s at-fence=%-8s resident=%-8s %s\n", $1, $2, $5, ($3==$6 ? "digest same" : "RE-DERIVED")}' \
| tee "${AAI9_ROOT}/parity/derived-activation-delta.txt"
echo "resident-after-activation: $(awk -F'\t' '{s+=$2} END {print s+0}' \
"${AAI9_ROOT}/parity/target-after-activation.tsv")"
Verified by: derived-activation-delta.txt names all five with both counts, and the resident total is recorded next to the fence total. The executed window’s readings, for comparison rather than as a target — a rebuild indexes the CURRENT repository corpus, so a different run legitimately produces different numbers: kb_chunk 10154 → 9055, kb_doc 707 → 659 (the delta entirely doc_type='markdown' 500 → 452, every other doc_type identical), and api_operation 2190, api_spec 1, doctor_search_projection 186 all re-derived at unchanged counts.
A LARGE derived drop is not automatically failure and not automatically fine. kb_doc fell 48 rows because the migrated snapshot had been indexed from an older repository state and the rebuild indexed the current one. That is correct behaviour. What would NOT be correct is a derived table arriving empty or near-empty, which means the rebuild failed rather than re-derived: check knowledge-rebuild.txt for the completion line before accepting any derived count, and treat a missing completion line as a stop condition.
Rollback is unaffected either way. The five are rebuildable on demand (KnowledgeSearchEndpoints.cs:102) and the frozen source still holds 10154/707 intact and retained until AAI-10, so a derived-table delta never justifies rolling back the move.
Confirm the Doctor grant persisted, then enable registration and prove the capability row (gate 8; the grant was reconciled live on 2026-08-22, registration remains off):
REPAIRED 2026-09-05 (sprint AAI-13, commit landed on
main; NOT YET REDEPLOYED). The two traps below described the overlay as it stood during the window.docker-compose.advisoryai.ymlnow mergesx-advisoryai-authority-identityon BOTH roles, so the web host has a host identity to ride and takes theifbranch; and it SETSDoctor__Registration__{PlatformBaseAddress, SelfEndpoint,Tenant}instead of commenting them, which is whatServiceDoctorRegistrationOptions’ own validation actually requires (ServiceDoctorRegistrar.cs:193-198). Correction to trap 1 while you are here:Doctor:Registration:Authorityis read ONLY by theelsebranch, and that branch requires Authority and ClientId and ClientSecret — so setting Authority alone would never have prevented the crash. Trap 2 was the whole cause. The threeelse-branch keys are deliberately absent and a conformance fact asserts their absence.Until the containers are recreated from the corrected overlay, both traps below still describe the RUNNING hosts — they name the window’s four
tmp/overlays in their owncom.docker.compose.project.config_files. Run the AAI-13 redeploy recipe first, then this flip.THIS FLIP CRASH-LOOPED THE HOST ON THE PRE-AAI-13 OVERLAY. Both traps as measured (both measured 2026-09-04, window
20260904-aai9; the attempt cost 16 restarts and left the estate with no serving AdvisoryAI web host until it was reverted).
Doctor:Registration:Authorityis required too, and the runbook named only the ClientId trap.RequireDoctorRegistrationValuedemandsAuthority,ClientIdandClientSecret(WebService/Program.cs:392-394, throwing at:433) — but only in theelsebranch taken whenadvisoryAuthorityEnabledis false (:361vs:381). The host fails closed at startup:System.InvalidOperationException: Doctor:Registration:Authority is required when registration is enabled, and must not be blank (fail closed).- The overlay makes that
elsebranch the live path, because it givesadvisoryai-webnoAdvisoryAi:Authorityblock at all.docker-compose.advisoryai.ymlsets all eightADVISORYAI__AdvisoryAi__Authority__*keys on the worker and none on the web host, while the predecessorstellaops-advisory-ai-webcarried nine (includingAudience). So “leaveClientIdunset and ride the host identity” has no host identity to ride here. The fix is to restore the web host’s block to predecessor parity — it is additive, since that section configures only the outbound client-credentials identity (Program.cs:112-175→AddStellaOpsAuthClient) while inbound bearer validation is a separate path (an unauthenticated call still returns 401 afterwards).SOURCE FIXED 2026-09-05 (sprint AAI-11,
4380e13610): both hosts now setclientOptions.TokenEndpointfrom their own section, and AAI-13 supplies the compose value (https://router.stella-ops.local/connect/token) together with the CA bundle mount andSSL_CERT_FILEthat TLS endpoint needs — the key without the trust only moves the failure from ID2083 toUntrustedRoot. The paragraph below is the window’s measurement, kept because it is the baseline any post-redeploy comparison is made against.Even with both fixed, this criterion did not close in that window, and the blocker was pre-existing source, not configuration: the
AdvisoryAi:Authoritybranch never setclientOptions.TokenEndpoint, while the doctor-registrationelsebranch did (:398). Against a lab Authority serving HTTP discovery, the token request is refused with OpenIddict ID2083 ("This server only accepts HTTPS requests."), surfacing asHttpRequestException: ... 400 (Bad Request)thrown byEnsureSuccessStatusCodeon the token call, not on the registration POST. The same defect is the root cause of the family’s long-standing unified-search adapter fallbacks — the predecessor’s own pre-window log carries 1,289falling back to snapshotlines for each ofFindingsSearchAdapter,PolicySearchAdapterandVexSearchAdapter, with the full Authority block present. Closing gate 8 therefore depends on a code fix (setTokenEndpointin that branch), not on this step.
# after flipping ADVISORYAI_DOCTOR_REGISTRATION_ENABLED=true and recreating advisoryai-web
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d stellaops_platform -c \
"SELECT count(*) > 0 FROM platform.doctor_capabilities WHERE service_name ILIKE '%advisoryai%';"
Verified by: the query prints t. The column is service_name, not service — the latter was in this runbook until 2026-09-04 and fails with ERROR: column "service" does not exist. The real columns are service_name, check_id, category, contract_version, endpoint, registered_at, last_seen_at. Leave Doctor__Registration__ClientId unset — this host holds exactly one Authority client and setting a different id fails the host closed (docker-compose.advisoryai.yml, the Doctor__Registration__* block — :216-222 at HEAD; the :166-171 pointer here was stale by 50 lines, corrected 2026-09-04. Locate it with grep -n 'Doctor__Registration' devops/compose/docker-compose.advisoryai.yml).
11. Confirm the auto-published surface, then flip the Console
CORRECTED 2026-09-04 (F-AAI9-15). This step used to say “Apply the route swap and flip the Console” and told the operator to run
tools/scripts/aai5-swap-advisoryai-routes.ps1three times, asserting all three exit 0. DO NOT DO THAT. The owner ruling of 2026-08-28 (“all microservices needs to autopublish no router configuration for microservices endpoints”), executed by SPRINT_20260809_001 RAR-581e276c2a3and RAR-78bebf33263, retired all seven routes the swap replaced, plus/apiand the^/api/v1/([^/]+)(.*)wildcard the OpsMemory surface rode. A static route out-ranks publication, so inserting^/api/advisoryai/v1(.*)would take the consolidated plane back from the auto-published layer — a regression, not a no-op. The script is now a hard refusal: every path exits 2. Under this runbook’sset -euo pipefailthe old text would have aborted the window here — after the restore, after parity, with the successors live and the legacy hosts stopped. See sprint AAI-9 gate 6 and D-AAI5-20.
There is no route to add. The consolidated surface is served by publication: the host maps native /api/advisoryai/v1/** endpoints (src/AdvisoryAI/StellaOps.AdvisoryAI.WebService/Endpoints/AdvisoryAiConsolidatedRouteAliases.cs) and the bridge discovers them from the host’s own EndpointDataSource (AspNetCoreEndpointDiscoveryProvider.DiscoverAspNetEndpoints). They go live with the successor image started in step 10 — which is why this step comes after it, not before.
PUBLICATION ALONE IS NOT ENOUGH: THE OVERLAY OMITS THE ROUTER ANCHOR, AND WITHOUT IT EVERY AdvisoryAI PATH RETURNS 404
TargetService=(none)(measured 2026-09-04, window20260904-aai9— this is the defect that would otherwise have failed gate 6).docker-compose.advisoryai.ymlis standalone, so it cannot referencex-router-microservice-defaultsindocker-compose.stella-services.yml:55-81, and the successor hosts start 22Router__keys short of the predecessor — includingRouter__Messaging__HeartbeatInterval,Router__Gateways__0__{Host,Port,TransportType},Router__Messaging__valkey__*,Router__Frame__Envelope__*andRouter__AuthorizationTrustMode.The symptom is deceptive: the host logs
Connected to router ... registered 175 endpoints, so it looks correctly published — but it never heartbeats, and the gateway builds its route table from a heartbeat-triggered metadata replay (Heartbeat discovered unknown messaging connection ... awaiting endpoint replay→Requested endpoint metadata replay ... gateway-state-miss→Endpoint metadata replay received from <service>). No heartbeat ⇒ no replay ⇒ no route. Restarting the service and the gateway does not fix it; the gateway logged replays from 28 other services and none fromadvisoryai.DONE 2026-09-05 (sprint AAI-13), with one correction: the anchor is inlined for the WEB host only.
docker-compose.advisoryai.ymlnow carriesx-advisoryai-router-defaults— 22 of the estate anchor’s 24 keys verbatim, the two mounted-mode keys left to the service’s own values, plusRouter__AuthorizationTrustMode: GatewayEnforced(the estate rolls that out keyed by PREDECESSOR names, andStellaRouterOptionsBasedefaults toHybrid). Secrets stay by variable reference.advisoryai-workergets NONE of them and that is deliberate:StellaOps.AdvisoryAI.Worker/Program.cscalls noAddRouterMicroserviceand the predecessoradvisory-ai-workermerged no router anchor, so the 22 keys the window set on it were inert. A conformance fact asserts the positive on web and the negative on worker; the expected set is read FROMx-router-microservice-defaults, so a key added there fails the AdvisoryAI suite. The running containers still predate this repair — confirm it by the gateway’s own log after the AAI-13 redeploy:HELLO received from advisoryai/...followed byEndpoint metadata replay received from advisoryai/..., before believing any 200.
Confirm the end state rather than swapping. The proof is a real request through the gateway, not a config diff:
# 1. No static route claims the namespace, on any of the THREE config surfaces.
for config in src/Router/StellaOps.Gateway.WebService/appsettings.json \
devops/compose/router-gateway-local.json \
devops/release/bundle/config/router-gateway.json; do
printf '%s advisoryai=%s opsmemory=%s\n' "${config}" \
"$(grep -c -e advisoryai -e advisory-ai "${config}")" \
"$(grep -c opsmemory "${config}")"
done | tee "${AAI9_ROOT}/activation/route-config-zero.txt"
! grep -qv 'advisoryai=0 opsmemory=0$' "${AAI9_ROOT}/activation/route-config-zero.txt"
# 2. The regression gate, run before AND after this step.
pwsh ./tools/scripts/test-targeted-xunit.ps1 \
-Project src/Router/__Tests/StellaOps.Gateway.WebService.Tests/StellaOps.Gateway.WebService.Tests.csproj \
-Class 'StellaOps.Gateway.WebService.Tests.Configuration.AdvisoryAiConsolidatedRouteConfigTests' \
| tee "${AAI9_ROOT}/activation/route-guard-advisoryai.txt"
pwsh ./tools/scripts/test-targeted-xunit.ps1 \
-Project src/Router/__Tests/StellaOps.Gateway.WebService.Tests/StellaOps.Gateway.WebService.Tests.csproj \
-Class 'StellaOps.Gateway.WebService.Tests.Configuration.RetiredRouteRegistryTests' \
| tee "${AAI9_ROOT}/activation/route-guard-retired.txt"
Verified by: every row of route-config-zero.txt reads advisoryai=0 opsmemory=0; AdvisoryAiConsolidatedRouteConfigTests reports Total: 15, Failed: 0 and RetiredRouteRegistryTests Total: 13, Failed: 0; and an authenticated request through the gateway to a consolidated path returns the host’s own answer with its expected status code. docker exec ... psql proves nothing about routing — the forcing function here is the HTTP response.
Two constraints survive from AAI-5, and only two:
- Do not regenerate
router-gateway-local.jsonblindly — F-AAI5-13 measured that thecatalog-changesroute ishttpsin the compose config andhttpinappsettings.jsonwith no override to upgrade it, so a regeneration silently DOWNGRADES a working Authority route. Diff for it and refuse a regeneration that flips it. - The manifest
tools/scripts/aai5-advisoryai-route-swap.jsonis kept for the route-table findings this window still needs (the downgrade above, the unlisted third config surface, and the single-frame transport ceiling on evidence-pack export). Read itsretirementblock first: it names which of its own claims are now false. Never feed it to the retired swap script.
Retired from this step, do not reinstate: the /api/v1/advisories coordination with the 006/008 owner (F-AAI5-5) — RAR-7 retired that row on 2026-09-02 and both programmes had already ruled it dies without a successor; and “expect the measured red set of 4 of 206 Configuration cases”, which described a swap that no longer runs against a table that no longer exists (the suite is 15 cases).
Then flip the six Console prefixes in api-path-prefixes.ts and rebuild/deploy the dist. Flip all six together, and only now: the native aliases exist only in the successor image, so an early flip is a 404 for as long as the predecessor is serving. Read the comment block above those values with care — it still describes the retired swap (sprint F-AAI9-16); the six values are correct.
Verified by: the built dist contains /api/advisoryai/v1 and no legacy AdvisoryAI prefix, and a browser session reaches the AdvisoryAI pages without a 404.
12. Live verification before declaring the move
Run every one of these against the migrated data, not a synthetic fixture:
| Forcing function | Verified by |
|---|---|
| Direct service health and auth | /health/live and /health/ready return 200 on both containers; an unauthenticated call to a protected route returns 401/403 |
| Scope enforcement on the lifted surface (add this row — it failed in the executed window) | A token carrying only openid profilemust be refused on a scope-gated route. Measured 2026-09-04: it returned 200 on GET /api/v1/opsmemory/decisions, /api/v1/opsmemory/stats, /api/advisoryai/v1/opsmemory/decisions and /v1/advisory-ai/runs. See the regression note below |
| Advisory draft against the migrated KB — NAMED LIMIT, not runnable in the 2026-09-04 lab | a draft generates and cites chunks that exist in the target kb_chunk. Judge it against the hub consensus projection, not raw advisory text — AAI-LD changed the worker’s document source. 2026-09-07: the executed window could not run this leg at all — chat is disabled on both predecessor and successor (ADVISORYAI__AdvisoryAI__Chat__Enabled=false) and no model-gateway container exists in that estate, so no draft can be generated there at any effort. The owner ruled Q-34 option A on 2026-09-07: AAI-9 is accepted with this leg carried as a named limit. An operator running this procedure in an estate that does have a chat-enabled host with a model gateway must run it; one that does not records the same limit rather than reading the empty row as a defect. The load-bearing half of the same forcing function — search through the application’s own path against the migrated kb_chunk, cited ids resolving in the target, a fabricated id returning 0 — is not covered by this limit and must pass |
| OpsMemory recall | a recall answers through the in-process provider on the consolidated host, with IOpsMemoryClient gone (D-AAI9-4). Measured 2026-09-10 (AAI-5): a decision recorded through the gateway (POST /api/advisoryai/v1/opsmemory/decisions, 201) was recalled by id (200) and by query (?cveId=, 1 row) on the same path. The similarity recall (GET /suggestions) answered analyzedRecords 0 even after an outcome, because API-recorded decisions carried an EMPTY similarity_vector; fixed in the record endpoint the same day (RecordedDecisions_CarryASimilarityVector_…). On an estate running an image older than that fix, run the suggestion leg only for chat-recorded decisions, or read the empty answer as this defect rather than as a missing recall |
| Evidence pack create → get → run → export | persists across a real host restart, and the export runs through the actual Router Microservice transport, not direct HTTP to a Router-disabled scratch host (F-AAI5-6). Measured 2026-09-10 (AAI-5): a 27 MiB pack (430 evidence items, 40 claims) created through the live gateway in 8.1 s (201), read back (200, 28.4 MB in 4.0 s), listed by its run, and exported as ONE buffered Router response of 28,426,544 bytes in 1.9 s (200) with every payload byte equal to the request; anonymous 401, scope-less token 403; the same bearer token direct to the host answered 401 (GatewayEnforced). The gateway’s Kestrel default caps request bodies at 30,000,000 bytes and surfaces the overflow as a 500, below the configured 100 MB routing limit (F-AAI5-22), so keep a create body under ~28 MiB or raise the Kestrel limit first |
| Four-persona gate | the recorded AAI-5/AAI-6 persona set passes |
| Error-log soak | zero permission errors and zero does not exist SQL errors across the soak window. A function similarity(…) does not exist line is finding D1 reappearing |
REGRESSION FOUND BY THIS STEP, 2026-09-04: THE CONSOLIDATION SILENTLY DROPPED SCOPE ENFORCEMENT ON THE LIFTED OPSMEMORY SURFACE, AND THE GATEWAY FAILS OPEN. A token carrying only
openid profilereturned 200 onGET /api/v1/opsmemory/decisionsand/stats; the same path on the standaloneopsmemory-webreturned 403 (requiredClaimType=scope, requiredClaimValue=ops-memory:read, service=opsmemory) a few hours earlier the same day. Unauthenticated still returns 401, so authentication is enforced and authorization is not. Positive control at the same moment, same token: Platform returned 403 with the gateway logginguser lacks claim scope=ops.health— so the gateway can enforce and simply has nothing to enforce for this family.Mechanism, and why the consolidation is what triggers it:
AuthorizationMiddleware.GetEffectiveClaims(endpoint.ServiceName, method, path)(:59-62) keys requirements by service name. AAI-9 moves these endpoints from serviceopsmemoryto serviceadvisoryai, so entries registered underopsmemoryno longer match — and:88-91then calls_next(context)wheneffectiveClaims.Count == 0, i.e. it fails open. The host does not re-check its ownRequireAuthorization(OpsMemoryPolicies.Read)becauseRouter__AuthorizationTrustMode=GatewayEnforcedtells it to trust the gateway. (A partial guard exists for state-changing methods, which inherit sibling-method claims at:70-86; GETs have no such backstop.) Verified not a configuration slip: the successor carriesRouter__OnMissingAuthorization=FailandRouter__AuthorizationTrustMode=GatewayEnforcedbyte-identical to both predecessors. (Router__OnMissingAuthorizationwas deleted estate-wide on 2026-09-07 by ROA-1(b): it bound to no property, so it had never selected any behaviour. The comparison above still holds — the successor and the predecessors were identical in it — it just compared a key that did nothing.)Consequence for FF-2: the write-then-recall still proves the data path (the row lands in
stellaops_advisoryaiwhile the frozen source stays at 0), but it does not prove the authorization posture — it would pass with a scope-less token. Run the negative control above alongside it, or FF-2 carries a vacuity it cannot see.
Verified by: each row’s own receipt under activation/. The soak is the step that catches D1, because the trigram fallback path only runs on queries that miss the FTS index.
13. Revoke source access and flip the ownership matrix
Only after step 12 is clean, and only for the predecessor identities — not for the schemas themselves, which stay intact for rollback retention until AAI-10.
THERE IS NO PREDECESSOR SERVICE IDENTITY TO REVOKE.
grants-forward.sqlIS AN ASSERTION, NOT A REVOCATION — DO NOT READ A COMPLETED STEP 13 AS “A BOUNDARY NOW EXISTS.” Measured 2026-09-04, window20260904-aai9:
Probe Result role_table_grantsonadvisoryai/opsmemorystellaopsonly — 182 and 14 grantspg_namespace.nspaclfor both schemasNULL(no explicit schema grants)pg_default_aclfor both schemasno rows roles matching %advis%/%opsmem%exactly one: advisoryaihas_database_privilege('advisoryai','stellaops_platform','CONNECT')f— already revokedThe three predecessor containers connect as the cluster superuser
stellaops, which also owns every object in both schemas and is the identity every other service onstellaops_platformdepends on. Revoking from it would be ineffective (ownership carries implicit privileges) and an estate-wide outage. The predecessors’ write fence is therefore the container stop in step 5, not a grant change, and the successor’s isolation is enforced from the other side and proven independently:probe-database-isolation.sh(augmented env, 18 roles × 18 databases) reportsadvisoryaiCONNECTED tostellaops_advisoryaiand refused on all 17 siblings.Write
grants-forward.sqlas a fail-loud assertion of that state (noCONNECTonstellaops_platformorstellaops_findings) andgrants-rollback.sqlas a no-op that verifies the source privileges are intact. Both were reviewed and dry-run green in the executed window.
docker exec "${AAI9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_SOURCE_DB}" \
-f "${AAI9_ROOT}/preflight/grants-forward.sql" \
| tee "${AAI9_ROOT}/activation/source-revoke.txt"
docker exec "${AAI9_PG_CONTAINER}" psql -X -qAt -F '|' -v ON_ERROR_STOP=1 \
-U "${AAI9_PG_OPERATOR}" -d "${AAI9_SOURCE_DB}" -c \
"SELECT grantee, table_schema, count(*)
FROM information_schema.role_table_grants
WHERE table_schema IN ('advisoryai','opsmemory')
GROUP BY 1,2 ORDER BY 1,2;" \
| tee "${AAI9_ROOT}/activation/source-grants-after.txt"
Verified by: source-grants-after.txt shows no predecessor service identity retaining access, and a soak of at least the recorded duration produces zero permission errors in any service log.
REPORT TWO NUMBERS, NEVER ONE (AAI-12). The window’s headline result is not “N rows migrated and verified”. It is:
- Rows copied and verified at the fence — the step 9b total across all 25 carried tables, with its invariant/derived subtotals. This is a copy-time proof and it is exact.
- Rows resident after activation — the step 10d total, with the five derived tables’ counts named individually and the invariant-set diff recorded as empty.
A single figure is stale or misleading depending on when it is read, and the two differ by design: the executed window recorded 13,238 at the fence and 12,091 resident, because the derived index was rebuilt from the current repository corpus. Quoting only the first is how a future operator re-counts, finds a smaller number, and concludes a successful move failed.
Then flip the four AdvisoryAI rows in docs/architecture/database-ownership-matrix.md to stellaops_advisoryai, in the same change that records both parity numbers in the sprint:
| Row | Location at HEAD (2026-09-04) |
|---|---|
advisory-ai-web + advisory-ai-worker (+ data-init) — wave table | :59 |
opsmemory-web — wave table | :60 |
advisory-ai-web + advisory-ai-worker — full rename map | :191 |
opsmemory-web — full rename map | :192 |
Locate them by content, never by line: grep -n 'advisory-ai-web\|opsmemory-web' docs/architecture/database-ownership-matrix.md. The two rename-map rows already name stellaops_advisoryai as the target; what the flip changes is the two wave-table cells, which still read advisoryai / opsmemory. The scanner wave row — “scanner — live on stellaops_scanner since 2026-08-29”, wave “2 — DONE” — is the format precedent. (Corrected 2026-09-04: this paragraph said “three rows” and omitted :60; the sprint gate 12’s :42,43,159,160 pointers were stale and are fixed there too.)
Keep the legacy schemas, image IDs, config blobs and stopped containers recoverable until AAI-10’s separately approved destructive retirement.
Rollback — preserve the freeze until the named unfreeze point
Enter rollback on any failed gate or any failed live verification. Do not reverse-copy target rows into the intact frozen source, and do not restore the recovery dump over that intact source.
Stop
stellaops-advisoryai-webandstellaops-advisoryai-workerand keep them stopped. Keep the three predecessor containers stopped too — the freeze is not lifted by entering rollback.Revert the Console prefixes from the recorded rollback plan. Restore the exact predecessor grants from
preflight/grants-rollback.sql. There is no route swap to revert (F-AAI9-15, 2026-09-04): the swap was retired by RAR-7 and never applied, so the gateway holds no static AdvisoryAI group to roll back. If step 11’s auto-published surface is the thing that failed, that is a publication problem on advisoryai-web, not a gateway edit — do not hand-add a static entry to “restore” the plane.Recreate only the three recorded predecessor service keys, with their recorded Compose file chains and image IDs from
rollback/*.image.txt— never a sibling service’s chain and never a floating tag. The rollback environment must pin the predecessor connection aliases back tostellaops_platform; reusing the successor overlay is NO-GO, because the legacy images carry the predecessor migration lineage and would apply it to whatever database they are given.Render that exact rollback chain into the ignored receipt and assert, without printing any connection string, that the predecessor aliases name the source database and that each service resolves to its recorded image ID.
RENDER EACH SERVICE ON ITS OWN CHAIN, AND PRODUCE THE RENDER AT STEP 2 — NOT HERE. Two defects measured 2026-09-04 in window
20260904-aai9:- The artifact this step asserts against did not exist. Step 2 captures
.image.txt,.config-files.txt,.environment.sha256,.mounts.jsonand.networks.json— no rendered Compose JSON. Nothing in the runbook createdrollback/predecessor-compose-render.json, so this step was unperformable as written. Render it at step 2, while the predecessors are still running: a rollback recipe first exercised after the fence cannot protect the window, and the chains includetmp/-resident files a sweep can remove. - A COMBINED render silently downgrades two of the three pins to floating TAGS, which contradicts step 3’s “never a floating tag”. The three chains are heterogeneous (18 / 15 / 10 files) and each ends in its own ID pin. Rendering all three on
advisory-ai-web’s 18-file superset chain — the natural thing to do, and what a single combined receipt file invites — resolvedadvisory-ai-workertostellaops/advisory-ai-worker:devandopsmemory-webtostellaops/opsmemory-web:bl2-9796c887-20260724. Rendering each on its own chain resolved both to their recorded image IDs. This failure is LATENT, which is why it must be asserted rather than eyeballed: both tags currently still resolve to the correct IDs, so the wrong method succeeds and looks right. If either tag moves — andadvisoryai-worker:devdiffers fromadvisory-ai-worker:devby one hyphen, while the nightlydocker image prune -afchurns tags — the rollback silently starts the wrong image.
# One render PER SERVICE, on that service's own recorded chain. for pair in stellaops-advisory-ai-web:advisory-ai-web \ stellaops-advisory-ai-worker:advisory-ai-worker \ stellaops-opsmemory-web:opsmemory-web; do container="${pair%%:*}"; service="${pair##*:}" mapfile -t files < <(tr ',' '\n' < "${AAI9_ROOT}/rollback/${container}.config-files.txt") args=(); for f in "${files[@]}"; do [ -n "$f" ] && args+=(-f "$f"); done docker compose -p stellaops --project-directory devops/compose \ --env-file devops/compose/.env "${args[@]}" config --format json \ > "${AAI9_ROOT}/rollback/${container}.render.json" want="$(awk '{print $2}' "${AAI9_ROOT}/rollback/${container}.image.txt")" got="$(jq -r --arg s "${service}" '.services[$s].image' \ "${AAI9_ROOT}/rollback/${container}.render.json")" test "${want}" = "${got}" || { echo "NO-GO: ${service} renders ${got}, not the recorded ${want}" >&2; exit 1; } jq -e --arg s "${service}" ' .services[$s].environment.ConnectionStrings__Default | type == "string" and test("(^|;)Database=stellaops_platform(;|$)"; "i") ' "${AAI9_ROOT}/rollback/${container}.render.json" > /dev/null doneVerified by: all three image comparisons are exact string matches against the step-2 receipts, and all three
jq -eDSN assertions exit 0. Both halves were run green in window20260904-aai9; the image half is the one that catches the latent tag trap above.- The artifact this step asserts against did not exist. Step 2 captures
Start the three predecessor images. Confirm their image IDs equal the step 2 receipts and all three health checks are green. (No route swap is in play — see step 2.)
Re-run
aai9_fingerprintsagainststellaops_platform; it must byte-matchparity/source-a.tsv. A difference is a database recovery incident, not permission to merge target rows back.Run the authenticated positive and negative predecessor probes from the approved activation record. The negative probe must prove unauthenticated and cross-tenant access remain refused.
UNFREEZE POINT
The write freeze may be lifted only after all five conditions are true at once:
- both successor containers are stopped;
- routes, Console prefixes, connection configuration and grants all resolve to the predecessor source;
- all three predecessor images are healthy and match the captured image IDs;
- the source’s canonical fingerprints still equal
parity/source-a.tsv; and - the authenticated positive plus unauthorized/cross-tenant negative probes pass.
Only then restart the predecessor writers and lift maintenance mode. Retain stellaops_advisoryai unchanged for investigation — do not drop it; a converged target with a failed parity proof is the primary evidence. If the source is damaged, keep every writer stopped and invoke the separately approved migration recovery or backup-restore procedure; this runbook does not authorize a destructive restore.
