Notify NTF-9 own-database data move and rollback

Audience: Notify implementer, database operator, live-deploy operator, rollback owner, and the owner approving the SPRINT_20260722_015 NTF-9 maintenance window.

Purpose: converge the empty stellaops_notify target from the successor migrations, freeze the two predecessor writers, copy only the 32 shared logical tables, prove exact row and sequence/identity parity, and preserve a tested rollback path before any route, flag, grant, or source-retirement action.

This is an operator procedure, not authorization to run it. It was prepared from source; it is not an execution receipt. Stop at the first failed assertion. Do not improvise a whole-schema restore, copy either migration ledger, copy notify.locks, or unfreeze a writer while authority is ambiguous.

Non-negotiable boundaries

Source-derived copy contract

The list below is the exact intersection of:

  1. surviving logical tables created by the active predecessor migrations; and
  2. logical tables created by the successor’s active 001–004 migrations.

NotifyDatabaseMoveRunbookConformanceTests derives both sets from the SQL and rejects drift in either direction. deliveries is named once as the logical partition root; the dump command uses --table-and-children and --load-via-partition-root, so every source partition is copied exactly once through the target root.

NTF9_COPY_TABLES=(
  anomaly_subscriptions
  channels
  correlation_runtime_incidents
  correlation_runtime_throttle_events
  dead_letter_entries
  deliveries
  digests
  escalation_policies
  escalation_states
  fallback_runtime_chains
  fallback_runtime_delivery_states
  inbox
  incident_report_timeline_states
  incidents
  localization_bundles
  maintenance_windows
  on_call_schedules
  operator_overrides
  pack_approvals
  quiet_hours
  retention_cleanup_executions_runtime
  retention_policies_runtime
  rules
  storm_runtime_events
  storm_runtime_states
  templates
  tenant_cross_grants
  tenant_isolation_violations
  tenant_resource_ownership
  throttle_configs
  webhook_security_configs
  webhook_validation_nonces
)

NTF9_TARGET_ONLY_TABLES=(
  nis2_csirt_signing_due
  nis2_csirt_signing_work
  nis2_incident_ledger_handoffs
)

Explicit exclusions:

Migration 004_nis2_tenant_stream_partition.sql changes no table topology. On a fresh target it is an intentional no-op until P6 exists, because there can be no original global-stream rows. If an upgrade target does contain notify.nis2-incident-ledger rows, keep the writer freeze and require the migration’s transactional receipt rekey. Its different epoch failure is a NO-GO: leave both source and target intact, do not mark 004 applied, and enter rollback rather than merging epochs.

Required approvals and real receipts

The window is NO-GO until every row names a real reviewed artifact. Blank, TBD, verbal, or source-only evidence is a failure.

Required inputReceipt
Window start/end, owner, DB operator, rollback ownerapproved change/window ID
Reviewed source and target image buildsimmutable digest plus provenance for both target roles
Source compositioncommit containing ce3ed472dcd20f662ddbf53e15b7df1dd33de9d5; targeted full-successor descriptor proof green (one Notify, Eventing, Catalog.Replication and credential-store migration host)
Database provisionstellaops_notify, role notify, NOSUPERUSER NOBYPASSRLS, sibling CONNECT revocations
Backupfull source dump SHA-256 plus isolated restore-test receipt using the same PostgreSQL major
Rollbackpredecessor image IDs, exact compose config_files, environment/config blob digests, forward and rollback grant plans
Credentialsexplicit zero-reference proof or a separately approved referenced-row copy/parity plan
Activation dependenciesFindings consumer/checkpoint/retention approval, Authority tenant feed proof, Doctor grant/capability-row proof, route/flag plan

1. Establish the workspace

Use opaque non-secret identifiers.

set -euo pipefail

export NTF9_WINDOW_ID='<approved-window-id>'
export NTF9_ROOT="tmp/ntf9-${NTF9_WINDOW_ID}"
export NTF9_PG_CONTAINER='stellaops-postgres'
export NTF9_PG_OPERATOR='stellaops'
export NTF9_SOURCE_DB='stellaops_platform'
export NTF9_TARGET_DB='stellaops_notify'
export NTF9_TARGET_ROLE='notify'

mkdir -p "${NTF9_ROOT}"/{preflight,backup,parity,rollback,activation}
test -n "${NTF9_WINDOW_ID}"
git rev-parse HEAD | tee "${NTF9_ROOT}/preflight/repository-head.txt"
git status --short | tee "${NTF9_ROOT}/preflight/repository-status.txt"

Explain every status row; never clean or reset the checkout. Attach the required approvals under preflight/, then prove they are non-empty:

for receipt in \
  window-approval.txt backup-restore-proof.txt grants-forward.sql grants-rollback.sql \
  notify-web-image.txt notify-worker-image.txt; do
  test -s "${NTF9_ROOT}/preflight/${receipt}" || {
    echo "NO-GO: missing ${receipt}" >&2
    exit 1
  }
done
grep -Eq '@sha256:[0-9a-f]{64}$' "${NTF9_ROOT}/preflight/notify-web-image.txt"
grep -Eq '@sha256:[0-9a-f]{64}$' "${NTF9_ROOT}/preflight/notify-worker-image.txt"

2. Capture rollback topology before changing anything

Both predecessor writer containers must exist. The target worker may be absent or stopped. Record the exact image ID and Compose file chain; do not reconstruct a rollback from a sibling service.

for container in stellaops-notify-web stellaops-notifier-worker; do
  docker inspect "${container}" --format '{{.Config.Image}} {{.Image}}' \
    > "${NTF9_ROOT}/rollback/${container}.image.txt"
  docker inspect "${container}" \
    --format '{{index .Config.Labels "com.docker.compose.project.config_files"}}' \
    > "${NTF9_ROOT}/rollback/${container}.config-files.txt"
  docker inspect "${container}" --format '{{json .Config.Env}}' \
    | sha256sum > "${NTF9_ROOT}/rollback/${container}.environment.sha256"
  docker inspect "${container}" --format '{{json .Mounts}}' \
    > "${NTF9_ROOT}/rollback/${container}.mounts.json"
  docker inspect "${container}" --format '{{json .NetworkSettings.Networks}}' \
    > "${NTF9_ROOT}/rollback/${container}.networks.json"
done

docker ps --format '{{.Names}}\t{{.Image}}\t{{.Status}}' \
  | sort | tee "${NTF9_ROOT}/preflight/containers-before.tsv"

The environment receipt is a hash only because the raw environment may contain secrets. The protected configuration source itself stays in its approved secret store.

3. Provision and converge an empty target before the writer freeze

Load PGPASSWORD_SERVICE and the target connection from the approved secret store without echoing them. Provision through the repository helper, then start the digest-pinned default-off notify-worker only long enough to converge notify, notify_app, and eventing. Use the exact reviewed Compose chain from the NTF-8 deployment runbook; do not recreate notify-web yet.

test -n "${PGPASSWORD_SERVICE:-}" || {
  echo 'NO-GO: PGPASSWORD_SERVICE is not loaded' >&2
  exit 1
}

bash tools/scripts/deploy/postgres/provision-service-database.sh \
  "${NTF9_TARGET_DB}" "${NTF9_TARGET_ROLE}" \
  --container "${NTF9_PG_CONTAINER}" --superuser "${NTF9_PG_OPERATOR}" \
  | tee "${NTF9_ROOT}/preflight/provision-target.txt"
unset PGPASSWORD_SERVICE

Before starting it, the reviewed render must show the exact own-database variable and must not set any activation flag true. Follow notify-deploy-ntf8-staged-stack.mdwith the recorded digest pin, capture its rendered JSON, start notify-worker, wait healthy, capture logs, then stop it:

test -s "${NTF9_ROOT}/preflight/notify-compose-render.json"
jq -e '
  .services["notify-worker"].environment.STELLAOPS_POSTGRES_NOTIFY_CONNECTION != null and
  (.services["notify-worker"].environment["Notify__DeliveryPipeline__Enabled"] // "false") == "false" and
  (.services["notify-worker"].environment["Notify__Nis2LedgerOutbox__Enabled"] // "false") == "false" and
  (.services["notify-worker"].environment.Catalog__Replication__Tenants__Enabled // "false") == "false" and
  (.services["notify-worker"].environment.Catalog__Replication__EnvironmentState__Enabled // "false") == "false"
' "${NTF9_ROOT}/preflight/notify-compose-render.json"

# Execute with the reviewed NTF-8 Compose array, not a reconstructed shorthand:
docker compose '<recorded -p/--project-directory/--env-file/-f chain>' up -d --no-deps notify-worker
until test "$(docker inspect -f '{{.State.Health.Status}}' stellaops-notify-worker)" = healthy; do sleep 2; done
docker logs stellaops-notify-worker > "${NTF9_ROOT}/preflight/notify-worker-converge.log" 2>&1
docker stop stellaops-notify-worker

The angle-bracket argument is deliberately not executable until the operator substitutes the exact reviewed chain and records it in the window receipt. A guessed chain is NO-GO.

Verify the target is empty, owns only its successor migration history, and has no retired table:

NTF9_TABLE_SQL="$(printf "'%s'," "${NTF9_COPY_TABLES[@]}")"
NTF9_TABLE_SQL="${NTF9_TABLE_SQL%,}"

docker exec -i "${NTF9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${NTF9_PG_OPERATOR}" -d "${NTF9_TARGET_DB}" -At <<SQL \
  | tee "${NTF9_ROOT}/preflight/target-empty-counts.tsv"
SELECT format('SELECT %L, count(*) FROM notify.%I;', relname, relname)
  FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
 WHERE n.nspname = 'notify'
   AND c.relkind IN ('r','p')
   AND NOT c.relispartition
   AND c.relname IN (${NTF9_TABLE_SQL})
 ORDER BY c.relname
\gexec
SQL
test "$(wc -l < "${NTF9_ROOT}/preflight/target-empty-counts.tsv")" \
  -eq "${#NTF9_COPY_TABLES[@]}"
awk -F '|' '$2 != 0 { exit 1 }' "${NTF9_ROOT}/preflight/target-empty-counts.tsv"

docker exec -i "${NTF9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${NTF9_PG_OPERATOR}" -d "${NTF9_TARGET_DB}" -At <<'SQL' \
  | tee "${NTF9_ROOT}/preflight/target-lineage.txt"
SELECT to_regclass('notify.locks') IS NULL;
SELECT migration_name FROM notify.schema_migrations ORDER BY migration_name;
-- FLAGGED 2026-08-28: this relation does NOT exist on a correctly converged target, so this
-- line ERRORS rather than reporting 0. Left in place, not silently deleted, because the
-- question it was written to answer -- did anything reconcile migrations behind our back? --
-- is still worth asking. Establish where that ledger was meant to come from, then either
-- restore the relation or replace this assertion with one that holds. Do not just drop it.
SELECT to_regclass('notify.schema_migration_reconciliations') IS NULL;
SELECT count(*) FROM notify.nis2_csirt_signing_due;
SELECT count(*) FROM notify.nis2_csirt_signing_work;
SELECT count(*) FROM notify.nis2_incident_ledger_handoffs;
SQL
grep -Fxq 't' "${NTF9_ROOT}/preflight/target-lineage.txt"
for migration in \
  001_notify_consolidated_baseline.sql \
  002_nis2_csirt_signing_work.sql \
  003_nis2_incident_ledger_handoff_idempotency.sql; do
  grep -Fxq "${migration}" "${NTF9_ROOT}/preflight/target-lineage.txt"
done
test "$(grep -Fxc '0' "${NTF9_ROOT}/preflight/target-lineage.txt")" -eq 4
! grep -Fq '001_v1_notify_baseline.sql' "${NTF9_ROOT}/preflight/target-lineage.txt"

Also require target role rolsuper=false, rolbypassrls=false, and PUBLIC CONNECT=false:

docker exec -i "${NTF9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${NTF9_PG_OPERATOR}" -d postgres -c \
  "SELECT d.datname, d.datdba::regrole AS owner,
          has_database_privilege('PUBLIC', d.datname, 'CONNECT') AS public_connect,
          r.rolsuper, r.rolbypassrls
     FROM pg_database d JOIN pg_roles r ON r.oid = d.datdba
    WHERE d.datname = '${NTF9_TARGET_DB}';" \
  | tee "${NTF9_ROOT}/preflight/target-owner-posture.txt"

4. Capture a full recovery backup

This backup includes predecessor schema and migration history because it is for disaster recovery, not for target restore. The isolated restore-test receipt must already be green.

docker exec -i "${NTF9_PG_CONTAINER}" pg_dump \
  -U "${NTF9_PG_OPERATOR}" -d "${NTF9_SOURCE_DB}" -Fc \
  --schema=notify --schema=notify_app \
  > "${NTF9_ROOT}/backup/source-notify-full.dump"
test -s "${NTF9_ROOT}/backup/source-notify-full.dump"
sha256sum "${NTF9_ROOT}/backup/source-notify-full.dump" \
  | tee "${NTF9_ROOT}/backup/source-notify-full.dump.sha256"
pg_restore --list "${NTF9_ROOT}/backup/source-notify-full.dump" \
  > "${NTF9_ROOT}/backup/source-notify-full.list"
grep -q 'SCHEMA.*notify' "${NTF9_ROOT}/backup/source-notify-full.list"
grep -q 'SCHEMA.*notify_app' "${NTF9_ROOT}/backup/source-notify-full.list"

Creating a dump is not a restore test. Version/command drift from the approved restore receipt is NO-GO.

5. Fence both predecessor writers and freeze the source

Stop, but do not remove, the live web writer and delivery consumer. The target worker remains stopped. Record sessions and obtain the DB operator’s signed no-writer verdict.

docker stop stellaops-notify-web stellaops-notifier-worker \
  | tee "${NTF9_ROOT}/preflight/writer-fence-stop.txt"

for container in stellaops-notify-web stellaops-notifier-worker stellaops-notify-worker; do
  if docker inspect "${container}" >/dev/null 2>&1; then
    test "$(docker inspect -f '{{.State.Running}}' "${container}")" = false
  fi
done

docker exec -i "${NTF9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${NTF9_PG_OPERATOR}" -d "${NTF9_SOURCE_DB}" -c \
  "SELECT pid, usename, application_name, client_addr, state, query_start
     FROM pg_stat_activity
    WHERE datname = '${NTF9_SOURCE_DB}'
    ORDER BY pid;" \
  | tee "${NTF9_ROOT}/preflight/source-sessions-after-fence.txt"

Build canonical fingerprints for all logical roots and capture them twice. deliveries includes every partition because a SELECT from the root includes its descendants.

ntf9_fingerprints() {
  local database="$1" output="$2" table count digest
  : > "${output}"
  for table in "${NTF9_COPY_TABLES[@]}"; do
    count="$(docker exec "${NTF9_PG_CONTAINER}" psql -XqAt -v ON_ERROR_STOP=1 \
      -U "${NTF9_PG_OPERATOR}" -d "${database}" \
      -c "SELECT count(*) FROM notify.${table};")"
    digest="$(docker exec "${NTF9_PG_CONTAINER}" psql -XqAt -v ON_ERROR_STOP=1 \
      -U "${NTF9_PG_OPERATOR}" -d "${database}" \
      -c "COPY (SELECT to_jsonb(t)::text FROM notify.${table} AS t
                 ORDER BY to_jsonb(t)::text COLLATE \"C\") TO STDOUT" \
      | sha256sum | awk '{print $1}')"
    printf '%s\t%s\t%s\n' "${table}" "${count}" "${digest}" >> "${output}"
  done
}

ntf9_fingerprints "${NTF9_SOURCE_DB}" "${NTF9_ROOT}/parity/source-a.tsv"
sleep 30
ntf9_fingerprints "${NTF9_SOURCE_DB}" "${NTF9_ROOT}/parity/source-b.tsv"
cmp "${NTF9_ROOT}/parity/source-a.tsv" "${NTF9_ROOT}/parity/source-b.tsv"

Any difference means the writer fence is incomplete. Do not copy.

6. Pin sequence and identity ownership before the dump

Derive every sequence owned by a copied table through pg_depend. Compare source and target ownership before restore. This list is currently empty because the predecessor’s only BIGSERIAL belonged to the dropped notify.audit, but the procedure must fail safely if a future migration adds one.

ntf9_sequence_inventory() {
  local database="$1" output="$2"
  docker exec -i "${NTF9_PG_CONTAINER}" psql -XqAt -F $'\t' -v ON_ERROR_STOP=1 \
    -U "${NTF9_PG_OPERATOR}" -d "${database}" -c \
    "SELECT format('%I.%I', sn.nspname, s.relname),
            format('%I.%I', tn.nspname, t.relname),
            a.attname,
            CASE a.attidentity WHEN 'a' THEN 'identity-always'
                               WHEN 'd' THEN 'identity-default'
                               ELSE 'owned-sequence' END
       FROM pg_class s
       JOIN pg_namespace sn ON sn.oid = s.relnamespace
       JOIN pg_depend d ON d.classid = 'pg_class'::regclass
                       AND d.objid = s.oid
                       AND d.refclassid = 'pg_class'::regclass
                       AND d.deptype IN ('a','i')
       JOIN pg_class t ON t.oid = d.refobjid
       JOIN pg_namespace tn ON tn.oid = t.relnamespace
       JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = d.refobjsubid
      WHERE s.relkind = 'S'
        AND sn.nspname = 'notify'
        AND tn.nspname = 'notify'
        AND t.relname IN (${NTF9_TABLE_SQL})
      ORDER BY 1,2,3;" > "${output}"
}

ntf9_sequence_inventory "${NTF9_SOURCE_DB}" "${NTF9_ROOT}/parity/source-sequences.tsv"
ntf9_sequence_inventory "${NTF9_TARGET_DB}" "${NTF9_ROOT}/parity/target-sequences.before.tsv"
cmp "${NTF9_ROOT}/parity/source-sequences.tsv" \
    "${NTF9_ROOT}/parity/target-sequences.before.tsv"

A non-empty matching inventory is supported below. A missing, extra, renamed, or differently owned sequence is NO-GO; do not accept row parity without resolving it.

7. Create the fenced data-only dump

Build exact table arguments from the source-derived list. deliveries selects all child partitions; the dump rewrites them through the partition root. Explicit exclusions are retained as a safety belt and as an auditable statement of intent.

NTF9_DUMP_ARGS=()
for table in "${NTF9_COPY_TABLES[@]}"; do
  if test "${table}" = deliveries; then
    NTF9_DUMP_ARGS+=(--table-and-children=notify.deliveries)
  else
    NTF9_DUMP_ARGS+=(--table="notify.${table}")
  fi
done
while IFS=$'\t' read -r sequence _; do
  test -n "${sequence}" && NTF9_DUMP_ARGS+=(--table="${sequence}")
done < "${NTF9_ROOT}/parity/source-sequences.tsv"

docker exec -i "${NTF9_PG_CONTAINER}" pg_dump \
  -U "${NTF9_PG_OPERATOR}" -d "${NTF9_SOURCE_DB}" -Fc \
  --data-only --load-via-partition-root --no-owner --no-privileges \
  --exclude-table=notify.locks \
  --exclude-table=notify.schema_migrations \
  --exclude-table=notify.schema_migration_reconciliations \
  "${NTF9_DUMP_ARGS[@]}" \
  > "${NTF9_ROOT}/backup/source-notify-fenced-data.dump"

test -s "${NTF9_ROOT}/backup/source-notify-fenced-data.dump"
sha256sum "${NTF9_ROOT}/backup/source-notify-fenced-data.dump" \
  | tee "${NTF9_ROOT}/backup/source-notify-fenced-data.dump.sha256"
pg_restore --list "${NTF9_ROOT}/backup/source-notify-fenced-data.dump" \
  > "${NTF9_ROOT}/backup/source-notify-fenced-data.list"
! grep -Eq 'TABLE DATA notify (locks|schema_migrations|schema_migration_reconciliations)' \
  "${NTF9_ROOT}/backup/source-notify-fenced-data.list"

8. Restore once, atomically, into the empty target

All three writer roles remain stopped. Do not truncate or merge a non-empty target. The restore is data-only and one transaction; any error leaves the converged empty target unchanged.

for container in stellaops-notify-web stellaops-notifier-worker stellaops-notify-worker; do
  if docker inspect "${container}" >/dev/null 2>&1; then
    test "$(docker inspect -f '{{.State.Running}}' "${container}")" = false
  fi
done

docker exec -i "${NTF9_PG_CONTAINER}" pg_restore \
  -U "${NTF9_PG_OPERATOR}" -d "${NTF9_TARGET_DB}" \
  --single-transaction --exit-on-error --data-only --no-owner --no-privileges \
  < "${NTF9_ROOT}/backup/source-notify-fenced-data.dump" \
  2>&1 | tee "${NTF9_ROOT}/backup/target-restore.txt"

Do not retry over a partially changed target. --single-transaction is the prerequisite that makes one retry against the still-empty target safe after the failure is diagnosed.

9. Prove exhaustive parity before any source retirement

Re-read the source after the copy and require it to equal the frozen receipt, then compare all 32 target roots exactly.

ntf9_fingerprints "${NTF9_SOURCE_DB}" "${NTF9_ROOT}/parity/source-after.tsv"
cmp "${NTF9_ROOT}/parity/source-a.tsv" "${NTF9_ROOT}/parity/source-after.tsv"

ntf9_fingerprints "${NTF9_TARGET_DB}" "${NTF9_ROOT}/parity/target-after.tsv"
diff -u "${NTF9_ROOT}/parity/source-after.tsv" \
        "${NTF9_ROOT}/parity/target-after.tsv" \
  | tee "${NTF9_ROOT}/parity/table-parity.diff"
test ! -s "${NTF9_ROOT}/parity/table-parity.diff"

Compare sequence state including the PostgreSQL is_called edge (which decides whether the next value is last_value or last_value + 1):

ntf9_sequence_state() {
  local database="$1" inventory="$2" output="$3" sequence
  : > "${output}"
  while IFS=$'\t' read -r sequence _; do
    test -z "${sequence}" && continue
    printf '%s\t' "${sequence}" >> "${output}"
    docker exec "${NTF9_PG_CONTAINER}" psql -XqAt -F $'\t' -v ON_ERROR_STOP=1 \
      -U "${NTF9_PG_OPERATOR}" -d "${database}" \
      -c "SELECT last_value, is_called FROM ${sequence};" >> "${output}"
  done < "${inventory}"
}

ntf9_sequence_inventory "${NTF9_TARGET_DB}" "${NTF9_ROOT}/parity/target-sequences.after.tsv"
cmp "${NTF9_ROOT}/parity/source-sequences.tsv" \
    "${NTF9_ROOT}/parity/target-sequences.after.tsv"
ntf9_sequence_state "${NTF9_SOURCE_DB}" "${NTF9_ROOT}/parity/source-sequences.tsv" \
  "${NTF9_ROOT}/parity/source-sequence-state.tsv"
ntf9_sequence_state "${NTF9_TARGET_DB}" "${NTF9_ROOT}/parity/target-sequences.after.tsv" \
  "${NTF9_ROOT}/parity/target-sequence-state.tsv"
cmp "${NTF9_ROOT}/parity/source-sequence-state.tsv" \
    "${NTF9_ROOT}/parity/target-sequence-state.tsv"

Pin the lineages and target-only emptiness again:

docker exec -i "${NTF9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${NTF9_PG_OPERATOR}" -d "${NTF9_TARGET_DB}" -At <<'SQL' \
  | tee "${NTF9_ROOT}/parity/non-copy-target-state.txt"
SELECT to_regclass('notify.locks') IS NULL;
SELECT count(*) FROM notify.schema_migration_reconciliations;
SELECT count(*) FROM notify.nis2_csirt_signing_due;
SELECT count(*) FROM notify.nis2_csirt_signing_work;
SELECT count(*) FROM notify.nis2_incident_ledger_handoffs;
SELECT count(*) FROM notify.schema_migrations
 WHERE migration_name NOT IN (
   '001_notify_consolidated_baseline.sql',
   '002_nis2_csirt_signing_work.sql',
   '003_nis2_incident_ledger_handoff_idempotency.sql',
   -- 004 added 2026-08-28: it SHIPS in
   -- src/Notify/__Libraries/StellaOps.Notify.Persistence.Consolidated/Migrations/ and is named in
   -- SPRINT_20260722_015 NTF-9's own NIS2 criterion, but this allowlist predated it. A correctly
   -- converged target therefore FAILED this gate on a legitimate migration. If you add a
   -- consolidated migration, add it here in the same change -- an allowlist that lags the
   -- migration set turns a correct target into a false blocker.
   '004_nis2_tenant_stream_partition.sql');
SQL
grep -Fxq 't' "${NTF9_ROOT}/parity/non-copy-target-state.txt"
test "$(grep -Fxc '0' "${NTF9_ROOT}/parity/non-copy-target-state.txt")" -eq 5

Compare RLS/FORCE-RLS posture and the canonical policy definitions on every copied root. A source/target policy mismatch is NO-GO even if the operator account can read both. Grant changes are not inferred here; they remain exclusively in the separately reviewed forward/rollback plans:

for database in "${NTF9_SOURCE_DB}" "${NTF9_TARGET_DB}"; do
  docker exec -i "${NTF9_PG_CONTAINER}" psql -XqAt -F $'\t' -v ON_ERROR_STOP=1 \
    -U "${NTF9_PG_OPERATOR}" -d "${database}" -c \
    "SELECT c.relname, c.relrowsecurity, c.relforcerowsecurity
       FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
      WHERE n.nspname = 'notify'
        AND c.relkind IN ('r','p')
        AND NOT c.relispartition
        AND c.relname IN (${NTF9_TABLE_SQL})
      ORDER BY c.relname;" \
    > "${NTF9_ROOT}/parity/${database}-rls.tsv"
done
cmp "${NTF9_ROOT}/parity/${NTF9_SOURCE_DB}-rls.tsv" \
    "${NTF9_ROOT}/parity/${NTF9_TARGET_DB}-rls.tsv"

for database in "${NTF9_SOURCE_DB}" "${NTF9_TARGET_DB}"; do
  docker exec -i "${NTF9_PG_CONTAINER}" psql -XqAt -F $'\t' -v ON_ERROR_STOP=1 \
    -U "${NTF9_PG_OPERATOR}" -d "${database}" -c \
    "SELECT c.relname,
            p.polname,
            p.polcmd,
            p.polpermissive,
            COALESCE((
              SELECT string_agg(
                       CASE WHEN role_oid.oid = 0 THEN 'PUBLIC' ELSE quote_ident(r.rolname) END,
                       ',' ORDER BY CASE WHEN role_oid.oid = 0 THEN 'PUBLIC' ELSE r.rolname END)
                FROM unnest(p.polroles) AS role_oid(oid)
                LEFT JOIN pg_roles r ON r.oid = role_oid.oid
            ), ''),
            COALESCE(pg_get_expr(p.polqual, p.polrelid, false), ''),
            COALESCE(pg_get_expr(p.polwithcheck, p.polrelid, false), '')
       FROM pg_policy p
       JOIN pg_class c ON c.oid = p.polrelid
       JOIN pg_namespace n ON n.oid = c.relnamespace
      WHERE n.nspname = 'notify'
        AND c.relname IN (${NTF9_TABLE_SQL})
      ORDER BY c.relname, p.polname, p.polcmd, p.polpermissive;" \
    > "${NTF9_ROOT}/parity/${database}-policies.tsv"
done
cmp "${NTF9_ROOT}/parity/${NTF9_SOURCE_DB}-policies.tsv" \
    "${NTF9_ROOT}/parity/${NTF9_TARGET_DB}-policies.tsv"

10. Prove builtin credential disposition

The copied Notify rows may contain builtin:// references whose ciphertext rows live in the separate crypto.secret_store lineage. This runbook does not guess or bulk-copy that schema. Count the exact source surfaces recognized by NotifyChannelSecretSlots:

docker exec -i "${NTF9_PG_CONTAINER}" psql -XqAt -F $'\t' -v ON_ERROR_STOP=1 \
  -U "${NTF9_PG_OPERATOR}" -d "${NTF9_SOURCE_DB}" <<'SQL' \
  | tee "${NTF9_ROOT}/parity/builtin-reference-counts.tsv"
SELECT 'channels', count(*)
  FROM notify.channels
 WHERE config::text LIKE '%builtin://%'
    OR COALESCE(credentials::text, '') LIKE '%builtin://%'
    OR metadata::text LIKE '%builtin://%';
SELECT 'webhook_security_configs', count(*)
  FROM notify.webhook_security_configs
 WHERE secret_key LIKE 'builtin://%';
SQL

If both counts are zero, record and approve the zero-reference proof. If either is non-zero, this window remains NO-GO until a separately reviewed plan identifies every referenced owner key and generation, copies only its required crypto.secret_store/KEK metadata rows, proves ciphertext byte parity and KEK identity, and successfully resolves each reference on the stopped target. A row count, shared master-key assumption, or whole-crypto dump is not a substitute.

Credential convergence attestation

The consolidated notify-web runtime is deliberately NOBYPASSRLS. It cannot run the predecessor’s web-only NotifyChannelSecretSealMigrationService: that service scans every tenant through an unkeyed connection with no app.tenant_id, while the default-off notify-worker convergence path never registers it. A target image therefore fails closed unless setup explicitly supplies Notify:SecretSealing:ExistingRowsConverged=true, and its startup error links back to this section. The flag is an admission receipt, not a developer default and not permission to grant BYPASSRLS.

Before setting it, prove that both the frozen source and copied target contain zero plaintext in the exact channel/webhook slots owned by NotifyChannelSecretSlots. The query prints only a count, never a value:

for database in "${NTF9_SOURCE_DB}" "${NTF9_TARGET_DB}"; do
  docker exec -i "${NTF9_PG_CONTAINER}" psql -XqAt -v ON_ERROR_STOP=1 \
    -U "${NTF9_PG_OPERATOR}" -d "${database}" <<'SQL' \
    > "${NTF9_ROOT}/parity/${database}-plaintext-secret-count.txt"
WITH secret_values AS (
  SELECT id::text AS row_id, config->>'secretRef' AS value
    FROM notify.channels
  UNION ALL
  SELECT c.id::text, property.value
    FROM notify.channels c
    CROSS JOIN LATERAL jsonb_each_text(
      COALESCE(c.config->'properties', '{}'::jsonb)) AS property
   WHERE property.key IN ('password', 'hmacSecret', 'apiKey', 'routingKey', 'botToken')
      OR property.key LIKE 'header.%'
      OR property.key LIKE 'notify.channel.nis2.header.%'
  UNION ALL
  SELECT id::text, metadata->'config'->>'secretRef'
    FROM notify.channels
  UNION ALL
  SELECT c.id::text, property.value
    FROM notify.channels c
    CROSS JOIN LATERAL jsonb_each_text(
      COALESCE(c.metadata->'config'->'properties', '{}'::jsonb)) AS property
   WHERE property.key IN ('password', 'hmacSecret', 'apiKey', 'routingKey', 'botToken')
      OR property.key LIKE 'header.%'
      OR property.key LIKE 'notify.channel.nis2.header.%'
  UNION ALL
  SELECT channel_id, secret_key
    FROM notify.webhook_security_configs
)
SELECT count(*)
  FROM secret_values
 WHERE value IS NOT NULL
   AND btrim(value) <> ''
   AND value !~* '^(builtin|vault|openbao|authref|file)://'
   AND value !~* '^legacy://';
SQL
  grep -Fxq '0' "${NTF9_ROOT}/parity/${database}-plaintext-secret-count.txt"
done

Then require the reviewed credential plan from the preceding gate to prove every referenced owner key/generation exists on the target, ciphertext plus nonce hashes match without exposing values, the target uses the identical KEK identity, and each stopped-target resolution succeeds. Record the four results in a non-secret receipt:

cat > "${NTF9_ROOT}/parity/NTF9_CREDENTIAL_CONVERGENCE_PASS.txt" <<'EOF'
source_plaintext=0
target_plaintext=0
credential_hash_parity=PASS
kek_identity=PASS
target_resolution=PASS
EOF
for assertion in \
  source_plaintext=0 target_plaintext=0 credential_hash_parity=PASS \
  kek_identity=PASS target_resolution=PASS; do
  grep -Fxq "${assertion}" \
    "${NTF9_ROOT}/parity/NTF9_CREDENTIAL_CONVERGENCE_PASS.txt"
done

Only after that receipt is non-empty may the reviewed activation environment set NOTIFY_SECRET_SEALING_EXISTING_ROWS_CONVERGED=true. Render the exact Compose chain offline and assert the target web host receives that value and explicitly disables both 6+2 compatibility template writers before starting it:

test -s "${NTF9_ROOT}/parity/NTF9_CREDENTIAL_CONVERGENCE_PASS.txt"
jq -e '
  .services["notify-web"].environment[
    "NOTIFY_NOTIFY__SECRETSEALING__EXISTINGROWSCONVERGED"] == "true" and
  .services["notify-web"].environment[
    "NOTIFY_NOTIFY__BOOTSTRAPTEMPLATESEEDING__ENABLED"] == "false"
' "${NTF9_ROOT}/preflight/notify-compose-render.json"

The explicit-off value belongs only to the parity-proven target overlay. Do not carry it into the four-alias predecessor rollback chain: predecessor compatibility keeps the default-on writers.

Retained-target fingerprint is an admission gate, not a resettable baseline

Immediately before any retained-target boot, re-run the same all-32-table and referenced-credential fingerprints and compare them with the preserved copy receipts. A mismatch is NO-GO even when its probable writer is known. Do not accept the current target as a new T0, repair it from the live source without a writer fence, reverse-copy, or disable bootstrap writers merely because eight templates still exist. Notify:BootstrapTemplateSeeding:Enabled=false is valid only while copied template parity is current.

The 2026-08-24 proof exercised this rule. The retained target’s preserved/current all-table hashes were 7f5faef56501b47d6e66b0f8d0e7498934e6266c00e7241ab0c134e98e6accd0 and 9cc1f5a21e6189f4de8aa253e549a79273064a37fa2e95295ed8597abcf82845. Only notify.templates differed: it still contained eight rows, while its canonical digest changed from f05aa0e07273f7ec0eb278b0e80946cb789e23cd9c4c222c97b2881da30f48d2 to 637c07540811c6aa0153af6ef0cbdd3fc5f828ae099c545bf787e69dad9067bd. The referenced credential remained exact at 45f4f8bbbf83e19cdc56f37be213a0314dfb6e0573d8ce853c81cb3023ef21b2. Therefore no candidate was started against stellaops_notify; no writer fence, target repair or reverse-copy ran.

The forward source contract was proved without weakening that gate on additive retained clone ntf9_notify_proof_20260824a. The clone inherited the current target exactly, remained owned by the notify role with PUBLIC CONNECT revoked and NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS, and had only Notify migrations 001-004 plus Crypto migrations 001,003,004. Production scratch image sha256:aa06cf190be4d3316f443735844f5291e2af7912b2d8c64b9c185f724bebf6f1 ran unrouted with no published port or Compose labels, all activation/consumer/Router/replica/audit/anomaly/Doctor gates false, existing-row convergence true and both 6+2 template writers false. Health was green with restart zero. T0/T1/T2 remained byte-identical for all 32 roots (9cc1f5a2...), the referenced credential (45f4f8bb...) and the exact ledgers (e81e2615...); logs had no legacy seal scan or predecessor baseline. The scratch container exited zero and was removed, while database clone and proof-only plugin volume ntf9_notify_proof_plugin_scratch_20260824a remain for audit. Both predecessors stayed healthy/restart-zero and the retained target’s pre/post fingerprint did not move.

This is admission evidence only. The image is scratch-only because its buildinfo records unpublished merge 5b6add0b2cca835ed620585c63990736159c33b9; do not promote it. Activation remains NO-GO until a fresh controlled window re-establishes current copied-template parity, repeats every gate above and builds the candidate from an exact published-main commit. An isolated clone proof never authorizes repoint, route/flag activation, grant revocation, forcing, soak, source retirement or deletion.

11. Declare the data-move gate, then continue the atomic window

Only after every check above is green may the DB operator create a real pass receipt:

{
  printf 'window=%s\n' "${NTF9_WINDOW_ID}"
  printf 'source=%s\n' "${NTF9_SOURCE_DB}"
  printf 'target=%s\n' "${NTF9_TARGET_DB}"
  printf 'logical_tables=%s\n' "${#NTF9_COPY_TABLES[@]}"
  printf 'table_parity=PASS\nsequence_identity_parity=PASS\n'
  printf 'locks_absent=PASS\nlegacy_ledgers_excluded=PASS\n'
  printf 'source_still_frozen=YES\n'
} > "${NTF9_ROOT}/parity/NTF9_DATA_MOVE_PASS.txt"
test -s "${NTF9_ROOT}/parity/NTF9_DATA_MOVE_PASS.txt"

This receipt does not claim activation, route swap, Findings consumption, credential parity when the count is non-zero, grant revocation, forcing, soak, or source retirement. Keep both predecessor writers stopped and continue the approved NTF-9 activation plan. The final delivery witness is notify-ntf9-delivery-forcing-function.md.

11.1 Activate and prove the Notify Doctor handshake

Source readiness does not prove the live handshake. Before enabling registration, reconcile the existing stellaops-notify Authority client so it holds platform:doctor:register, obtain a fresh client-credentials token as proof, and store a redacted receipt at preflight/authority-stellaops-notify-doctor-grant.txt. Do not record the client secret or bearer token. The Standard descriptor is desired-state input, not evidence that the live Authority row changed. Both tracked descriptor paths now include platform:doctor:register on stellaops-notify; confirm the exact reviewed source before reconciliation and do not substitute a one-off live grant that diverges from descriptor convergence.

The canonical NTF-9 activation reuses the Authority client already required by the tenant-catalog replica. Set NOTIFY_DOCTOR_REGISTRATION_ENABLED=true in the protected window environment and leave NOTIFY_DOCTOR_CLIENT_ID, NOTIFY_DOCTOR_AUTHORITY, and NOTIFY_DOCTOR_CLIENT_SECRET unset. A blank Doctor client id explicitly means “reuse the Notify host identity”; setting a different id makes the host fail closed rather than overwriting its one auth-client options object. Render the exact reviewed NTF-8 Compose chain, save it at activation/notify-compose-render.json, and prove the last-wins values before recreating notify-web:

test -s "${NTF9_ROOT}/preflight/authority-stellaops-notify-doctor-grant.txt"
test -s "${NTF9_ROOT}/activation/notify-compose-render.json"
jq -e '
  .services["notify-web"].environment["Catalog__Replication__Tenants__Enabled"] == "true" and
  .services["notify-web"].environment["Doctor__Registration__Enabled"] == "true" and
  (.services["notify-web"].environment["Doctor__Registration__ClientId"] // "") == "" and
  .services["notify-web"].environment["Doctor__Registration__SelfEndpoint"] ==
    "http://notify.stella-ops.local/doctor/notify-web/checks"
' "${NTF9_ROOT}/activation/notify-compose-render.json" >/dev/null

export NTF9_NOTIFY_WEB_RECREATE_SINCE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
# Execute with the reviewed NTF-8 Compose array; never reconstruct it from this shorthand.
docker compose '<recorded -p/--project-directory/--env-file/-f chain>' \
  up -d --force-recreate --no-deps notify-web
until test "$(docker inspect -f '{{.State.Health.Status}}' stellaops-notify-web)" = healthy; do
  sleep 2
done

As in section 3, the angle-bracket argument is deliberately non-executable until the operator substitutes the recorded chain from the approved window receipt.

After the digest-pinned host is healthy, use an in-memory token carrying only the approved ops.health read capability to prove both the service-owned surface and Platform’s registered capability rows. Five rows, an exact endpoint, and non-stale heartbeats are required; a 401/403, missing row, duplicate row, wrong endpoint, or stale row is an activation NO-GO.

read -r -s -p 'NTF-9 ops.health token: ' NTF9_OPS_HEALTH_TOKEN; printf '\n'
test -n "${NTF9_OPS_HEALTH_TOKEN}"

notify_doctor_code="$(curl -sk \
  -o "${NTF9_ROOT}/activation/notify-doctor-checks.json" -w '%{http_code}' \
  -H "Authorization: Bearer ${NTF9_OPS_HEALTH_TOKEN}" \
  https://notify.stella-ops.local/doctor/notify-web/checks)"
test "${notify_doctor_code}" = 200
jq -e '
  .service == "notify-web" and .contractVersion == "doctor-check/v1" and
  ([.checks[].checkId] | sort) == [
    "doctor.notify.deliveries.partition-coverage",
    "doctor.notify.rls-posture",
    "doctor.standard.db.connection",
    "doctor.standard.db.migration-status",
    "doctor.standard.db.size-budget"
  ]
' "${NTF9_ROOT}/activation/notify-doctor-checks.json" >/dev/null

platform_doctor_code="$(curl -sk \
  -o "${NTF9_ROOT}/activation/platform-notify-doctor-capabilities.json" -w '%{http_code}' \
  -H "Authorization: Bearer ${NTF9_OPS_HEALTH_TOKEN}" \
  https://platform.stella-ops.local/api/v1/platform/doctor/capabilities)"
test "${platform_doctor_code}" = 200
jq -e '[.capabilities[] | select(.serviceName == "notify-web")] as $notify |
  ($notify | length) == 5 and
  ($notify | all(
    .contractVersion == "doctor-check/v1" and
    .endpoint == "http://notify.stella-ops.local/doctor/notify-web/checks" and
    .stale == false))
' "${NTF9_ROOT}/activation/platform-notify-doctor-capabilities.json" >/dev/null

docker logs --since "${NTF9_NOTIFY_WEB_RECREATE_SINCE}" stellaops-notify-web 2>&1 \
  | grep -F 'Doctor capability catalog registered: notify-web, 5 check(s).' \
  | tee "${NTF9_ROOT}/activation/notify-doctor-registration.log"
unset NTF9_OPS_HEALTH_TOKEN

This receipt proves NTF-7 adoption and one NTF-9 activation dependency only. It does not replace the tenant checkpoint, delivery forcing pair, Findings retention/consumer proof, bounded soak, or source-retirement gates.

Rollback — preserve the freeze until the named unfreeze point

Enter rollback on any failed gate or activation/forcing failure. Do not reverse-copy target rows into the intact frozen source, and do not restore the disaster-recovery dump over that intact source.

  1. Keep stellaops-notify-web, stellaops-notifier-worker, and stellaops-notify-worker stopped.

  2. Restore the exact predecessor route/alias/configuration and grants from the reviewed rollback plan. Recreate only the recorded predecessor service keys with their recorded Compose file chains and image IDs. Do not use a sibling service’s chain or a floating tag. Before recreation, the protected rollback environment/override must pin all four old-image connection aliases to the captured source database: STELLAOPS_POSTGRES_NOTIFY_CONNECTION, NOTIFY_NOTIFY__STORAGE__CONNECTIONSTRING, NOTIFY_Postgres__Notify__ConnectionString, and Postgres__Notify__ConnectionString. Reusing the target activation environment is a NO-GO: the old image carries the predecessor migration lineage and will try to apply it to the successor ledger under FORCE RLS.

  3. Render that exact rollback chain into the ignored receipt and assert, without printing the connection strings, that every alias names stellaops_platform:

    jq -e '
  [
    .services["notify-web"].environment.STELLAOPS_POSTGRES_NOTIFY_CONNECTION,
    .services["notify-web"].environment.NOTIFY_NOTIFY__STORAGE__CONNECTIONSTRING,
    .services["notify-web"].environment.NOTIFY_Postgres__Notify__ConnectionString,
    .services["notify-web"].environment.Postgres__Notify__ConnectionString
  ] | all(type == "string" and test("(^|;)Database=stellaops_platform(;|$)"; "i"))
' "${NTF9_ROOT}/rollback/notify-web-compose-render.json"

  4. Start the predecessor images with delivery/tenant/NIS2 successor flags still false. Confirm their image IDs equal the pre-window receipts and both health checks are green.

  5. Re-run ntf9_fingerprints against stellaops_platform; it must byte-match parity/source-a.tsv. A difference is a database recovery incident, not permission to merge target rows back.

  6. Run the authenticated positive and negative predecessor probes from the approved activation record. The negative probe must prove unauthenticated/cross-tenant access remains refused.

UNFREEZE POINT

The maintenance/write freeze may be lifted only after all five conditions are true at once:

  1. both successor roles are stopped;
  2. route, aliases, connection configuration, and grants resolve to the predecessor source;
  3. both predecessor images are healthy and match the captured image IDs;
  4. the source’s canonical fingerprints still equal the frozen pre-copy receipt; and
  5. authenticated positive plus unauthorized/cross-tenant negative probes pass.

Only then restart/enable predecessor writers and lift maintenance mode. Retain the target database unchanged for investigation. If the source is damaged, keep every writer stopped and invoke the separately approved migration recovery / backup-restore procedure; this runbook does not authorize destructive restore.