Evidence EVD-9 data move and cutover

Audience: Evidence implementer, Database Operations, live-deploy operator, rollback owner, and the owner approving the SPRINT_20260722_011 EVD-9 maintenance window.

Purpose: move the live attestor, proofchain, and evidence_locker schemas from stellaops_platform into the Evidence family’s own stellaops_evidence database, prove canonical row parity and both integrity chains, start the digest-pinned successor roles, perform the one gateway/Console repoint, revoke predecessor access, and soak without dropping the source schemas.

ROUTE LEG RETIRED — 2026-09-04. The gateway route-swap parts of this procedure must not be run. The owner ruling of 2026-08-28 (“all microservices needs to autopublish no router configuration for microservices endpoints”) was executed by SPRINT_20260809_001 RAR-5/RAR-7 (81e276c2a3, 83bfc9359f, f4b1774cf6). All fifteen rows in tools/scripts/evd5-evidence-route-swap.json are gone from both route tables, which now hold 22 routes total; Evidence is served by the auto-published layer. A static gateway entry out-ranks publication, so reinstating those fifteen rows — by applying the EVD-5 swap, or by “reconciling” the manifest until -Check passes and then applying — would claw the Evidence plane back off the published layer: a regression, not a no-op. (Concretely today the scripts cannot do this on their own — with nothing to match they write nothing, then falsely report a successful apply and exit 0; see the window steps.)

Since ced8a2d286 pwsh tools/scripts/evd5-swap-evidence-routes.ps1 refuses on both invocations — -Check and a bare run each report the swap as RETIRED and exit 2, measured 2026-09-04 at HEAD. Section 7 step 3’s “exact fifteen predecessor routes” NO-GO gate is therefore unsatisfiable, and section 8 step 3’s apply no longer runs.

Do instead: confirm the auto-published layer serves /api/evidence/v1/** through the gateway (authorized 2xx, unauthenticated 401/403) and that the two Policy false friends (/api/v1/attestor/policies, /api/v1/attestor/reports) still resolve to Policy. Everything in this runbook that is not the route swap — the data move, both integrity chains, parity, revocation and soak — still stands. Model for the retired shape: offlinekit-cutover-ok6-route-swap.md.

This is an operator procedure, not authorization to execute it. EVD-5, EVD-8, and EVD-9 are one window. Stop at the first failed gate. Do not improvise a temporary build-boundary pin, a partial route swap, a second data copy, or a schema drop.

Non-negotiable boundaries

Scratch rehearsal namespace preflight (read-only)

Run this gate before every disposable Evidence up, including a repeat with a prefix that was used earlier. Compose can require a non-empty variable, but it cannot reject compose* or stellaops*, and -p does not replace explicit physical names. A collision is NO-GO: choose a new prefix. Do not delete or rename the resource that made the gate red; it may belong to the standing estate or an interrupted rehearsal.

The checker is read-only. Its self-test uses an injected inventory and does not contact Docker. The normal check rejects standing-name prefixes case-insensitively, then requires all exact derived web/worker/database/cache container names, Evidence/ProofChain/database/cache volume names, the network name, and every container/volume/network carrying the same Compose project label to be absent. The injected self-test drives each of those 14 exact name/project-label probes red in turn.

export EVIDENCE_SCRATCH_RESOURCE_PREFIX="evd8scratch-$(date -u +%Y%m%d%H%M%S)-$$"
export EVD8_SCRATCH_OVERRIDE="tmp/scratch/${EVIDENCE_SCRATCH_RESOURCE_PREFIX}/docker-compose.throwaway.yml"

pwsh tools/scripts/validate/check-evidence-scratch-resources.ps1 -SelfTest
pwsh tools/scripts/validate/check-evidence-scratch-resources.ps1 \
  -Prefix "${EVIDENCE_SCRATCH_RESOURCE_PREFIX}"

test -s "${EVD8_SCRATCH_OVERRIDE}"
: "${EVIDENCE_IMAGE_TAG:?build the two successor images under a unique scratch tag first}"
: "${EVIDENCE_SCRATCH_POSTGRES_PASSWORD:?load a scratch-only database password}"
: "${POSTGRES_IMAGE:?set an audited digest-pinned local PostgreSQL image}"
: "${VALKEY_IMAGE:?set an audited digest-pinned local Valkey image}"
: "${KESTREL_CERT_PASSWORD:?load the development certificate password from its local source}"
: "${STELLAOPS_DEFAULT_REGION:?set the installation residency sentinel for this rehearsal}"
: "${EVIDENCE_AUTHORITY_CLIENT_ID:?set the reviewed scratch Evidence client id}"
: "${EVIDENCE_AUTHORITY_CLIENT_SECRET:?load the scratch client secret}"
: "${STELLAOPS_EVIDENCELOCKER_CAPSULE_ED25519_SEED_B64:?load a scratch-only capsule signing seed}"
: "${STELLAOPS_IDENTITY_ENVELOPE_SIGNING_KEY:?load the estate identity-envelope signing key from its local source}"

# The base file interpolates before last-file replacement, so use an invalid non-secret sentinel.
export STELLAOPS_POSTGRES_EVIDENCE_CONNECTION='SCRATCH_OVERLAY_MUST_REPLACE_THIS'

EVD8_SCRATCH_COMPOSE=(
  -p "${EVIDENCE_SCRATCH_RESOURCE_PREFIX}"
  --env-file /dev/null
  -f devops/compose/docker-compose.evidence.yml
  -f "${EVD8_SCRATCH_OVERRIDE}"
  -f devops/compose/docker-compose.evidence.scratch.yml
)

docker compose "${EVD8_SCRATCH_COMPOSE[@]}" config --quiet

# Re-run immediately before creation; the earlier clean inventory is not a reservation.
pwsh tools/scripts/validate/check-evidence-scratch-resources.ps1 \
  -Prefix "${EVIDENCE_SCRATCH_RESOURCE_PREFIX}"
docker compose "${EVD8_SCRATCH_COMPOSE[@]}" up -d

The throwaway override must publish no ports, use the prefixed network only, and give its disposable database/cache containers the exact physical names <prefix>-db / <prefix>-cache and its disposable volumes <prefix>-postgres-data / <prefix>-cache-data. Those are the names the preflight proves absent. The checked-in last file labels every declared service, volume, and network with com.stellaops.scratch=true and com.stellaops.scratch.resource-prefix=<prefix> for attribution. Teardown only the exact scratch project, then reuse the same checker as the residue gate:

docker compose "${EVD8_SCRATCH_COMPOSE[@]}" down --volumes --remove-orphans
pwsh tools/scripts/validate/check-evidence-scratch-resources.ps1 \
  -Prefix "${EVIDENCE_SCRATCH_RESOURCE_PREFIX}"

This preflight authorizes no EVD-9 data move, route swap, predecessor stop, grant change, or live Compose activation. Those remain inside the owner-attended atomic window below.

Rehearsal record 2026-09-03 and the absent-predecessor variant

The procedure below was rehearsed on 2026-09-03 (SPRINT_20260722_011 EVD-5, receipt docs/implplan/_evidence/20260903-evd5-scratch-forcing-and-evd9-rehearsal.md) on an isolated scratch project with the live PostgreSQL used read-only. What it measured changes how the window must be planned on this estate:

Required approvals and receipts

The window is NO-GO until every field names a real, reviewed artifact. A blank value, TBD, stale report, or verbal substitution is a failure.

Required inputReceipt recorded before the window
Approved start/end, change owner, DB operator, gateway operator, Console operator, rollback ownerchange/window ID and approver
Current build-boundary report0 unpinned, 0 stale, 0 grown, 0 activation mismatches, 0 impure client SDKs
EVD-5 source ownershipcheck-evidence-p19-transition.ps1 -RequirePost reports the coherent post-state: the four Evidence-to-predecessor implementation pairs are gone without temporary pins and all three active predecessor host projects have left the supported source tree
Tenant and default-region resolutionlocal catalog-replica forcing test/commit for tenant slug-to-UUID; setup receipt naming the installation’s STELLAOPS_DEFAULT_REGION and the tested fail-closed source commit; both Authority Standard descriptors plus tests pin stellaops-evidence-web to default tenant, stellaops, client_credentials, and exactly catalog:replicate platform:doctor:register; live token/feed/checkpoint receipts contain no secret
Direct callers and SQL readersreviewed repoint manifest, every row closed or owner-excluded; its SQL-reader section must contain the exact two rows in consolidation design §4.3 and show both foreign database authorities removed
EVD-6 Console changetested commit and Console image digest; /policies and /reports remain Policy-owned
Target rolesweb/worker image digests, provenance receipts, runtime render, authenticated rehearsal. The render must carry the Attestor half the retired attestor definition carried (SPRINT_20260722_011 D-EVD5-17): attestor__signer__baseUrl, attestor__rekor__primary__{url,publicKeyBase64}, attestor__transparency__local__*, attestor__signing__keys__0__* with the PEM mount, attestor__verdictTrust__keys__0__*, attestor__security__signerIdentity__mode__*, attestor__security__mtls__requireClientCertificate, attestor__rekorQueue__enabled. Missing keys mean product defaults: mTLS-only writes and checkpoint_public_key_missing on every local-log entry (measured 2026-09-05, _evidence/20260905-evd9-attestor-half-carry-over.md). Assert with docker compose … config --format json before section 8, not after the forcing function fails.
Backupfull source-schema dump SHA-256 and isolated restore-test receipt
Permissionsreviewed forward and rollback grant plans; no ad-hoc role names
Rollbackpredecessor image IDs, exact compose lists, source config blobs, named restart commands
Soakduration, query owner, success thresholds, abort thresholds

The pre-window reviewer signs this table in the window record. That signature does not waive a live measurement below.

The initial source census found two direct SQL consumers. Sprint 021 PLT-1A closed Platform’s legacy-capsule erasure row, and sprint 026 CM-5 closed the CLI local-transparency row. The current production-source census therefore has zero foreign direct SQL readers of the carried schemas: Platform’s SQL adapter and silent no-refusal binding are deleted, while the CLI has no selectable PostgreSQL mode or local connection string and instead verifies the proof returned by the authenticated owner API. Canonical Evidence configuration wins at composition; invalid canonical configuration and owner failure fail closed without predecessor, database, or Rekor-v2 fallback.

Those source closures are not live cutover receipts. The window still captures the configured owner origin and proves each request crosses the selected predecessor or target route. The same receipt must enumerate and close the remaining direct HTTP bindings; a zero-row SQL census is not a substitute for that caller inventory. CM-5 also does not close the CLI’s seven remaining direct Attestor-family ProjectReference edges, which are source-boundary debt outside this database-reader repoint.

The reviewed HTTP inventory is consolidation-design.md §4.4. caller-repoints.tsv must carry every ID from EVD-HTTP-00 through EVD-HTTP-20, including rows classified as stopped, on-demand, dormant or unwired; those rows need an owner-signed exclusion and the command/config evidence that proves the classification at the window. Active rows need old and new origin, exact method/path, authentication mode, forced request ID, status and response digest. The source/compose guard EvidenceDirectHttpCallerConformanceTests must be green on the reviewed window commit; it reads the per-binding census in consolidation-design.md §4.5, so a binding this window repoints or retires is recorded there rather than in the test. Do not replace this with a hostname grep: several active callers inherit their old origin from source defaults, and CLI EvidenceCommandGroup uses the canonical Evidence variable with mixed legacy paths.

1. Establish the window workspace

Use opaque identifiers; do not put credentials in these variables.

set -euo pipefail

export EVD9_WINDOW_ID='<approved-window-id>'
export EVD9_ROOT="tmp/evd9-${EVD9_WINDOW_ID}"
export EVD9_PG_CONTAINER='stellaops-postgres'
export EVD9_PG_SUPERUSER='stellaops'
export EVD9_SOURCE_DB='stellaops_platform'
export EVD9_TARGET_DB='stellaops_evidence'
export EVD9_TARGET_ROLE='evidence'
export EVD9_PROJECT='stellaops'

mkdir -p "${EVD9_ROOT}"/{preflight,backup,parity,forcing,rollback,soak}
test -n "${EVD9_WINDOW_ID}"
git rev-parse HEAD | tee "${EVD9_ROOT}/preflight/repository-head.txt"
git status --short | tee "${EVD9_ROOT}/preflight/repository-status.txt"

Explain every status row before continuing. Do not clean or reset the shared checkout. The window change set must be a reviewed commit, not loose edits.

2. Run the fail-closed preflight

  1. Record the build-boundary verdict. The command must exit 0 with unpinned, stale, grown, activation mismatch, and impure client SDK all at zero.

    pwsh tools/scripts/build-boundary/generate-build-boundary-report.ps1 -Check \
  2>&1 | tee "${EVD9_ROOT}/preflight/build-boundary.txt"

  2. Prove the Evidence ownership transition is complete rather than hidden. The self-test first proves the checker rejects a mixed project set, a temporary pin, and a family reclassification while the supported predecessor hosts remain. -RequirePost must then report the coherent post-state. A coherent pre-state is still NO-GO; it needs the owner-approved EVD-9/EVD-10 transition decision, not a pin.

    pwsh tools/scripts/build-boundary/check-evidence-p19-transition.ps1 -SelfTest
pwsh tools/scripts/build-boundary/check-evidence-p19-transition.ps1 -RequirePost \
  2>&1 | tee "${EVD9_ROOT}/preflight/evidence-p19-transition.txt"

  3. Verify the route package without writing. -Check must report the exact fifteen predecessor routes, ten overlay keys to replace with one, both Policy false friends, and the Platform /api/v2/evidence route. Any other drift is NO-GO. Retired 2026-09-04 (banner above): the script now refuses and exits 2, and that is the correct end state, not a NO-GO to reconcile. Do not run it. Verify instead that the auto-published Evidence surfaces answer through the gateway and that the Policy false friends still resolve to Policy.

    # RETIRED 2026-09-04 -- do not run. The script reports the swap as retired and exits 2.
# pwsh tools/scripts/evd5-swap-evidence-routes.ps1 -Check
#   2>&1 | tee "${EVD9_ROOT}/preflight/route-swap-check.txt"

  4. Attach the reviewed caller/SQL-reader manifest, EVD-6 receipt, backup restore proof, image provenance, permission plans, approval, and Evidence tenant-replica identity receipts. Both checked-in Authority Standard descriptors now define stellaops-evidence-web, and their focused test pins its exact default tenant, stellaops audience, client_credentials grant, environment secret reference, and the two-scope set catalog:replicate platform:doctor:register. That is source preparation only. docker-compose.evidence.yml deliberately retains :? guards for the explicit client ID and operator-owned secret; the live row must be reconciled and token/feed probes must prove the intended claim set without recording the secret. This assertion rejects absent or empty files.

    for receipt in \
  caller-repoints.tsv console-evd6.txt backup-restore-proof.txt \
  authority-evidence-descriptors.txt authority-evidence-tests.txt \
  authority-evidence-token-claims.txt authority-evidence-feed.txt \
  evidence-web-image.txt evidence-worker-image.txt grants-forward.sql grants-rollback.sql \
  window-approval.txt; do
  test -s "${EVD9_ROOT}/preflight/${receipt}" || {
    echo "NO-GO: missing preflight receipt ${receipt}" >&2
    exit 1
  }
done

grep -Eq '@sha256:[0-9a-f]{64}$' "${EVD9_ROOT}/preflight/evidence-web-image.txt"
grep -Eq '@sha256:[0-9a-f]{64}$' "${EVD9_ROOT}/preflight/evidence-worker-image.txt"

  5. Re-measure the live topology. The expected serving writers are stellaops-attestor, stellaops-evidence-locker-web, and stellaops-evidence-locker-worker; tileproxy and TSA are recorded but not fenced. Neither target role may already be running. If a writer is absent (measured on 2026-09-03: all three, plus their images and both carried volumes), the docker inspect loop below fails for it; record the absence as the receipt for that container and run section 4 in the absent-predecessor variant described above.

    docker ps --format '{{.Names}}\t{{.Image}}\t{{.Status}}' \
  | sort | tee "${EVD9_ROOT}/preflight/containers-before.tsv"

for container in \
  stellaops-attestor stellaops-evidence-locker-web stellaops-evidence-locker-worker \
  stellaops-attestor-tileproxy stellaops-tsa; do
  docker inspect --format '{{index .Config.Labels "com.docker.compose.project.config_files"}}' \
    "${container}" > "${EVD9_ROOT}/rollback/${container}.config-files.txt"
  docker inspect --format '{{.Config.Image}} {{.Image}}' "${container}" \
    > "${EVD9_ROOT}/rollback/${container}.image.txt"
  docker inspect --format '{{json .Mounts}}' "${container}" \
    > "${EVD9_ROOT}/rollback/${container}.mounts.json"
  docker inspect --format '{{json .NetworkSettings.Networks}}' "${container}" \
    > "${EVD9_ROOT}/rollback/${container}.networks.json"
done

test -z "$(docker ps -q --filter name='^/stellaops-evidence-web$')"
test -z "$(docker ps -q --filter name='^/stellaops-evidence-worker$')"

  6. Record database/role state and source estimates. Historical sprint counts are context, not acceptance values.

    docker exec -i "${EVD9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${EVD9_PG_SUPERUSER}" -d postgres -c \
  "SELECT datname, datdba::regrole AS owner, datallowconn FROM pg_database
    WHERE datname IN ('${EVD9_SOURCE_DB}','${EVD9_TARGET_DB}') ORDER BY datname;" \
  | tee "${EVD9_ROOT}/preflight/databases-before.txt"

docker exec -i "${EVD9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_SOURCE_DB}" -At -F $'\t' -c \
  "SELECT schemaname, relname, n_live_tup
     FROM pg_stat_user_tables
    WHERE schemaname IN ('attestor','proofchain','evidence_locker')
    ORDER BY schemaname, relname;" \
  | tee "${EVD9_ROOT}/preflight/source-estimated-counts.tsv"

Stand down if the target database already contains domain data, a target role is running, or any required receipt/checker is red.

3. Capture a restorable source backup

Take the full schema+data backup before the writer fence, then the exact transfer snapshot after the fence. The first is disaster recovery; the second is the frozen dataset to copy.

docker exec -i "${EVD9_PG_CONTAINER}" pg_dump \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_SOURCE_DB}" -Fc \
  --schema=attestor --schema=proofchain --schema=evidence_locker \
  --schema=evidence_locker_app \
  > "${EVD9_ROOT}/backup/source-domain-full.dump"

test -s "${EVD9_ROOT}/backup/source-domain-full.dump"
sha256sum "${EVD9_ROOT}/backup/source-domain-full.dump" \
  | tee "${EVD9_ROOT}/backup/source-domain-full.dump.sha256"
pg_restore --list "${EVD9_ROOT}/backup/source-domain-full.dump" \
  > "${EVD9_ROOT}/backup/source-domain-full.list"
grep -q 'SCHEMA.*attestor' "${EVD9_ROOT}/backup/source-domain-full.list"
grep -q 'SCHEMA.*proofchain' "${EVD9_ROOT}/backup/source-domain-full.list"
grep -q 'SCHEMA.*evidence_locker' "${EVD9_ROOT}/backup/source-domain-full.list"

evidence_locker_app is not optional (found 2026-09-05, window 20260905-evd9). Every evidence_locker RLS policy calls evidence_locker_app.require_current_tenant(), and pg_depend shows it is the only foreign schema the three families’ policies and functions reference. A dump of the three domain schemas alone loads its data and then fails at the first CREATE POLICYon a standalone restore (schema "evidence_locker_app" does not exist) - while every row count still matches, which is how the gap hides. The 2026-09-03 rehearsal never hit it because it restore-tested the fenced data-only dump into a converged target that already carried the functions. Restore-test the FULL dump into an empty database with --exit-on-error and require exit 0, not just matching counts.

Compare PostgreSQL versions and command shape with the restore-tested receipt. A newly created dump has not itself been restored; a version or command drift requires a new isolated restore test.

4. Fence the writers and freeze the source

Only the three database-writer containers stop. Do not stop Signer, tileproxy, or TSA.

docker stop \
  stellaops-attestor \
  stellaops-evidence-locker-web \
  stellaops-evidence-locker-worker \
  | tee "${EVD9_ROOT}/preflight/writer-fence-stop.txt"

for container in stellaops-attestor stellaops-evidence-locker-web stellaops-evidence-locker-worker; do
  test "$(docker inspect -f '{{.State.Running}}' "${container}")" = 'false'
done

docker exec -i "${EVD9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_SOURCE_DB}" -c \
  "SELECT pid, usename, application_name, client_addr, state
     FROM pg_stat_activity WHERE datname = '${EVD9_SOURCE_DB}' ORDER BY pid;" \
  | tee "${EVD9_ROOT}/preflight/source-sessions-after-fence.txt"

The DB operator compares that list with the captured writer identities and signs that none remain. Then capture exact counts twice, at least 30 seconds apart. The files must be byte-identical.

snapshot_counts() {
  local database="$1" output="$2"
  docker exec -i "${EVD9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
    -U "${EVD9_PG_SUPERUSER}" -d "${database}" -At -F $'\t' <<'SQL' > "${output}"
SELECT format(
         'SELECT %L, count(*) FROM %I.%I;',
         schemaname || '.' || tablename, schemaname, tablename)
  FROM pg_tables
 WHERE schemaname IN ('attestor','proofchain','evidence_locker')
   AND tablename NOT IN ('schema_migrations','schema_migration_reconciliations')
 ORDER BY schemaname, tablename
\gexec
SQL
}

snapshot_counts "${EVD9_SOURCE_DB}" "${EVD9_ROOT}/parity/source-counts-a.tsv"
sleep 30
snapshot_counts "${EVD9_SOURCE_DB}" "${EVD9_ROOT}/parity/source-counts-b.tsv"
cmp "${EVD9_ROOT}/parity/source-counts-a.tsv" "${EVD9_ROOT}/parity/source-counts-b.tsv"

Create the frozen data-only transfer dump. Predecessor migration ledgers are excluded because the target’s fresh Evidence baseline remains its migration authority. Derive the ledger list from the source, do not copy it from here: the EvidenceLocker predecessor kept its ledger under the non-standard name evidence_locker.evidence_schema_version, which the 2026-09-05 window found only when the data-only restore died mid-way on a table the target never creates (the target was then partially loaded and had to be re-truncated). Any table present in the source but absent from the converged target is a ledger or residue to exclude, and the parity denominator is source tables - excluded ledgers (33 - 2 = 31 on that window), not a constant.

# Every source table the converged target does not have is the exclusion list: derived, not assumed.
docker exec -i "${EVD9_PG_CONTAINER}" psql -X -At -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_SOURCE_DB}" -c \
  "SELECT schemaname||'.'||tablename FROM pg_tables WHERE schemaname IN ('attestor','proofchain','evidence_locker') ORDER BY 1;" \
  > "${EVD9_ROOT}/parity/source-tables.txt"
# Compare with the target's table list captured in section 5; each source-only name becomes an --exclude-table below.

docker exec -i "${EVD9_PG_CONTAINER}" pg_dump \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_SOURCE_DB}" -Fc --data-only \
  --schema=attestor --schema=proofchain --schema=evidence_locker \
  --exclude-table=attestor.schema_migrations \
  --exclude-table=attestor.schema_migration_reconciliations \
  --exclude-table=proofchain.schema_migrations \
  --exclude-table=proofchain.schema_migration_reconciliations \
  --exclude-table=evidence_locker.schema_migrations \
  --exclude-table=evidence_locker.schema_migration_reconciliations \
  --exclude-table=evidence_locker.evidence_schema_version \
  > "${EVD9_ROOT}/backup/source-domain-fenced-data.dump"

test -s "${EVD9_ROOT}/backup/source-domain-fenced-data.dump"
sha256sum "${EVD9_ROOT}/backup/source-domain-fenced-data.dump" \
  | tee "${EVD9_ROOT}/backup/source-domain-fenced-data.dump.sha256"
pg_restore --list "${EVD9_ROOT}/backup/source-domain-fenced-data.dump" \
  > "${EVD9_ROOT}/backup/source-domain-fenced-data.list"

5. Provision and converge the target

Load the Evidence role password from the approved secret store into PGPASSWORD_SERVICE without echoing it, then use the repository’s idempotent provisioner.

test -n "${PGPASSWORD_SERVICE:-}" || {
  echo 'NO-GO: PGPASSWORD_SERVICE was not loaded from the approved secret store' >&2
  exit 1
}

bash tools/scripts/deploy/postgres/provision-service-database.sh \
  "${EVD9_TARGET_DB}" "${EVD9_TARGET_ROLE}" \
  --container "${EVD9_PG_CONTAINER}" --superuser "${EVD9_PG_SUPERUSER}" \
  | tee "${EVD9_ROOT}/preflight/provision-target.txt"
unset PGPASSWORD_SERVICE

Verify ownership/isolation before a target process starts:

docker exec -i "${EVD9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${EVD9_PG_SUPERUSER}" -d postgres -c \
  "SELECT d.datname, d.datdba::regrole AS owner,
          has_database_privilege('public', d.datname, 'CONNECT') AS public_connect,
          r.rolsuper, r.rolbypassrls
     FROM pg_database d JOIN pg_roles r ON r.oid = d.datdba
    WHERE d.datname = '${EVD9_TARGET_DB}';" \
  | tee "${EVD9_ROOT}/preflight/target-ownership.txt"

Acceptance is owner evidence, public_connect = false, rolsuper = false, and rolbypassrls = false.

Load the reviewed STELLAOPS_POSTGRES_EVIDENCE_CONNECTION, certificate password, Evidence service identity, installation region, and capsule signing seed into the process environment. Do not print them. Use only the Evidence overlay and reviewed digest-pin override, with --no-deps; do not reconstruct or recreate the live stack. These checks intentionally name the required inputs without printing their values.

test -s "${EVD9_ROOT}/preflight/evidence-digest-pin.override.yml"
test -n "${STELLAOPS_POSTGRES_EVIDENCE_CONNECTION:-}"
test -n "${KESTREL_CERT_PASSWORD:-}"
test -n "${STELLAOPS_DEFAULT_REGION:-}"
test -n "${EVIDENCE_AUTHORITY_CLIENT_ID:-}"
test -n "${EVIDENCE_AUTHORITY_CLIENT_SECRET:-}"
test -n "${STELLAOPS_EVIDENCELOCKER_CAPSULE_ED25519_SEED_B64:-}"
test -n "${STELLAOPS_IDENTITY_ENVELOPE_SIGNING_KEY:-}"

EVD9_COMPOSE=(
  -p "${EVD9_PROJECT}"
  --env-file devops/compose/.env
  -f devops/compose/docker-compose.evidence.yml
  -f "${EVD9_ROOT}/preflight/evidence-digest-pin.override.yml"
)

docker compose "${EVD9_COMPOSE[@]}" up -d --no-deps --force-recreate evidence-web
until docker inspect -f '{{.State.Health.Status}}' stellaops-evidence-web | grep -qx healthy; do
  sleep 2
done
docker logs stellaops-evidence-web > "${EVD9_ROOT}/preflight/evidence-web-converge.log" 2>&1
docker stop stellaops-evidence-web

Prove the baseline shape before clearing seed data. Expected shape is 10 Attestor tables, 8 ProofChain tables plus artifact_canonical_records, 13 EvidenceLocker tables, one fresh Evidence migration ledger, and 13 EvidenceLocker tables with both RLS and FORCE RLS. Any difference is NO-GO; do not edit a migration in the window.

docker exec -i "${EVD9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_TARGET_DB}" -c \
  "SELECT n.nspname AS schema_name, c.relkind, count(*)
     FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE n.nspname IN ('attestor','proofchain','evidence_locker','evidence')
      AND c.relkind IN ('r','m')
    GROUP BY n.nspname, c.relkind ORDER BY n.nspname, c.relkind;
   SELECT count(*) FILTER (WHERE relrowsecurity) AS rls,
          count(*) FILTER (WHERE relforcerowsecurity) AS forced
     FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE n.nspname = 'evidence_locker' AND c.relkind = 'r';" \
  | tee "${EVD9_ROOT}/preflight/target-baseline-shape.txt"

6. Copy the frozen data

This clears only the newly converged target domain tables. All target roles and predecessor writers must still be stopped.

for container in \
  stellaops-attestor stellaops-evidence-locker-web stellaops-evidence-locker-worker \
  stellaops-evidence-web stellaops-evidence-worker; do
  test -z "$(docker ps -q --filter name="^/${container}$")"
done

docker exec -i "${EVD9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_TARGET_DB}" <<'SQL'
SELECT format('TRUNCATE TABLE %I.%I CASCADE;', schemaname, tablename)
  FROM pg_tables
 WHERE schemaname IN ('attestor','proofchain','evidence_locker')
 ORDER BY schemaname, tablename
\gexec
SQL

docker exec -i "${EVD9_PG_CONTAINER}" pg_restore \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_TARGET_DB}" \
  --exit-on-error --data-only --no-owner --no-privileges \
  < "${EVD9_ROOT}/backup/source-domain-fenced-data.dump" \
  2>&1 | tee "${EVD9_ROOT}/backup/target-restore.txt"

docker exec -i "${EVD9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_TARGET_DB}" -c \
  'REFRESH MATERIALIZED VIEW proofchain.artifact_canonical_records;' \
  | tee "${EVD9_ROOT}/backup/target-materialized-view-refresh.txt"

7. Prove row parity and integrity

7.1 Per-table count and canonical-row SHA-256

For every ordinary table, compare count and SHA-256 of C-collated canonical JSONB rows. Migration ledgers are intentionally excluded. An empty table still has a stable digest.

mapfile -t EVD9_TABLES < <(
  docker exec -i "${EVD9_PG_CONTAINER}" psql -X -At \
    -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_SOURCE_DB}" -c \
    "SELECT format('%I.%I', schemaname, tablename)
       FROM pg_tables
      WHERE schemaname IN ('attestor','proofchain','evidence_locker')
        AND tablename NOT IN ('schema_migrations','schema_migration_reconciliations')
      ORDER BY schemaname, tablename;"
)

table_count() {
  local database="$1" table="$2"
  docker exec -i "${EVD9_PG_CONTAINER}" psql -X -At \
    -U "${EVD9_PG_SUPERUSER}" -d "${database}" -c "SELECT count(*) FROM ${table};"
}

table_digest() {
  local database="$1" table="$2"
  docker exec -i "${EVD9_PG_CONTAINER}" psql -X -qAt \
    -U "${EVD9_PG_SUPERUSER}" -d "${database}" -c \
    "COPY (SELECT to_jsonb(t)::text FROM ${table} AS t
            ORDER BY to_jsonb(t)::text COLLATE \"C\") TO STDOUT" \
    | sha256sum | awk '{print $1}'
}

printf 'table\tsource_count\ttarget_count\tsource_sha256\ttarget_sha256\tverdict\n' \
  > "${EVD9_ROOT}/parity/table-parity.tsv"

for table in "${EVD9_TABLES[@]}"; do
  source_count="$(table_count "${EVD9_SOURCE_DB}" "${table}")"
  target_count="$(table_count "${EVD9_TARGET_DB}" "${table}")"
  source_hash="$(table_digest "${EVD9_SOURCE_DB}" "${table}")"
  target_hash="$(table_digest "${EVD9_TARGET_DB}" "${table}")"
  verdict=PASS
  test "${source_count}" = "${target_count}" || verdict=FAIL
  test "${source_hash}" = "${target_hash}" || verdict=FAIL
  printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
    "${table}" "${source_count}" "${target_count}" "${source_hash}" "${target_hash}" "${verdict}" \
    | tee -a "${EVD9_ROOT}/parity/table-parity.tsv"
done

! grep -q $'\tFAIL$' "${EVD9_ROOT}/parity/table-parity.tsv"

Compare the refreshed materialized view separately. Both count/hash pairs must match; a stale source view is a finding, not permission to refresh the frozen source.

for database in "${EVD9_SOURCE_DB}" "${EVD9_TARGET_DB}"; do
  printf '%s\t%s\t%s\n' "${database}" \
    "$(table_count "${database}" proofchain.artifact_canonical_records)" \
    "$(table_digest "${database}" proofchain.artifact_canonical_records)"
done | tee "${EVD9_ROOT}/parity/artifact-canonical-records.tsv"

7.2 Exhaustive verdict-ledger linkage

The first row per tenant has a null prev_hash; every later row names the preceding verdict_hash. The result must be 0 on source and target.

for database in "${EVD9_SOURCE_DB}" "${EVD9_TARGET_DB}"; do
  docker exec -i "${EVD9_PG_CONTAINER}" psql -X -At -v ON_ERROR_STOP=1 \
    -U "${EVD9_PG_SUPERUSER}" -d "${database}" -c \
    "WITH ordered AS (
       SELECT tenant_id, verdict_hash, prev_hash,
              lag(verdict_hash) OVER (
                PARTITION BY tenant_id ORDER BY created_at, ledger_id) AS expected_prev,
              row_number() OVER (
                PARTITION BY tenant_id ORDER BY created_at, ledger_id) AS rn
         FROM attestor.verdict_ledger
     )
     SELECT count(*) FROM ordered
      WHERE (rn = 1 AND prev_hash IS NOT NULL)
         OR (rn > 1 AND prev_hash IS DISTINCT FROM expected_prev);"
done | tee "${EVD9_ROOT}/parity/verdict-chain-breaks.txt"

test "$(sort -u "${EVD9_ROOT}/parity/verdict-chain-breaks.txt" | tr -d '[:space:]')" = '0'

7.3 Exhaustive proof-spine recomputation

This gate mirrors DeterministicMerkleTreeBuilder: UTF-8 leaves, global bytewise lexical sort, SHA-256 per leaf, duplicate the final leaf hash to the next power of two, then SHA-256 of left || right until one lowercase 64-hex root remains. It checks every target spine against its stored bundle_id.

Create tmp/evd9-<id>/parity/verify-spines.py from this reviewed body:

import hashlib
import json
import sys

failures = 0
checked = 0

for raw in sys.stdin:
    if not raw.strip():
        continue
    row = json.loads(raw)
    leaves = [str(row["entry_id"])]
    leaves.extend(sorted(str(value) for value in row["evidence_ids"]))
    leaves.append(str(row["reasoning_id"]))
    leaves.append(str(row["vex_id"]))
    encoded = sorted(value.encode("utf-8") for value in leaves)
    level = [hashlib.sha256(value).digest() for value in encoded]
    target = 1
    while target < len(level):
        target <<= 1
    while len(level) < target:
        level.append(level[-1])
    while len(level) > 1:
        level = [
            hashlib.sha256(level[index] + level[index + 1]).digest()
            for index in range(0, len(level), 2)
        ]
    actual = level[0].hex()
    expected = str(row["bundle_id"]).lower()
    checked += 1
    if actual != expected:
        failures += 1
        print(f"FAIL\t{row['entry_id']}\t{expected}\t{actual}")

print(f"SUMMARY\tchecked={checked}\tfailures={failures}")
raise SystemExit(1 if failures else 0)

Run it and retain the output:

docker exec -i "${EVD9_PG_CONTAINER}" psql -X -qAt -v ON_ERROR_STOP=1 \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_TARGET_DB}" -c \
  "COPY (
     SELECT json_build_object(
       'entry_id', entry_id, 'evidence_ids', evidence_ids,
       'reasoning_id', reasoning_id, 'vex_id', vex_id, 'bundle_id', bundle_id)::text
       FROM proofchain.spines ORDER BY entry_id
   ) TO STDOUT" \
  | python "${EVD9_ROOT}/parity/verify-spines.py" \
  | tee "${EVD9_ROOT}/parity/proof-spines.txt"

7.4 Artifact foreign key

Count dangling evidence_artifacts.bundle_id values after the copy. A non-zero result is NO-GO and needs an owner disposition; do not add the constraint NOT VALID, because new inserts are still checked in that form.

orphan_count="$(docker exec -i "${EVD9_PG_CONTAINER}" psql -X -At -v ON_ERROR_STOP=1 \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_TARGET_DB}" -c \
  "SELECT count(*)
     FROM evidence_locker.evidence_artifacts a
     LEFT JOIN evidence_locker.evidence_bundles b ON b.bundle_id = a.bundle_id
    WHERE b.bundle_id IS NULL;")"
printf '%s\n' "${orphan_count}" | tee "${EVD9_ROOT}/parity/evidence-artifact-orphans.txt"
test "${orphan_count}" = '0'

docker exec -i "${EVD9_PG_CONTAINER}" psql -X -v ON_ERROR_STOP=1 \
  -U "${EVD9_PG_SUPERUSER}" -d "${EVD9_TARGET_DB}" <<'SQL'
DO $do$
BEGIN
  IF NOT EXISTS (
    SELECT 1 FROM pg_constraint
     WHERE conname = 'fk_artifacts_bundle'
       AND conrelid = 'evidence_locker.evidence_artifacts'::regclass
  ) THEN
    ALTER TABLE evidence_locker.evidence_artifacts
      ADD CONSTRAINT fk_artifacts_bundle
      FOREIGN KEY (bundle_id)
      REFERENCES evidence_locker.evidence_bundles(bundle_id)
      ON DELETE CASCADE;
  END IF;
END $do$;
SQL

Re-run the target table receipt after adding the constraint. Data counts/hashes must not change.

8. Start the successor and perform the one repoint

  1. Start both target roles from the same immutable override. Use --no-deps.

    docker compose "${EVD9_COMPOSE[@]}" up -d --no-deps --force-recreate \
  evidence-web evidence-worker

until docker inspect -f '{{.State.Health.Status}}' stellaops-evidence-web | grep -qx healthy; do
  sleep 2
done
test "$(docker inspect -f '{{.State.Running}}' stellaops-evidence-worker)" = 'true'
docker inspect --format '{{.Config.Image}} {{.Image}}' \
  stellaops-evidence-web stellaops-evidence-worker \
  | tee "${EVD9_ROOT}/forcing/target-images-running.txt"

  2. Probe the successor directly before exposing it. /health/ready must be 200; an anonymous scoped endpoint must be 401/403; an authorized read must succeed; all Evidence doctor checks must be healthy. Also prove current_database() is exactly stellaops_evidence, both catalog_replica.replica_rows and eventing.consumer_checkpoints exist there, the tenants consumer has a durable checkpoint, and a known active slug resolves while a measured unknown slug does not. Store results under forcing/; never record a token or secret.

  3. Apply the reviewed window change set containing all three inseparable changes:

    • the route script’s fifteen-to-one group in both route sources — retired; omit this part entirely (banner above). Nothing is applied to the route sources in this window;
    • deletion of the manifest’s ten old overlay keys plus ^/api/evidence/v1(.*) -> https://evidence.stella-ops.local/api/evidence/v1$1;
    • the EVD-6 Console prefix/image change, preserving Policy’s /api/v1/attestor/policies and /api/v1/attestor/reports routes.

    Do not hand-edit a partial state. Re-run the checker after the route and overlay change.

    # RETIRED 2026-09-04 -- do not run either command. Both invocations report the swap as
# retired and exit 2 since ced8a2d286; neither writes anything.
# pwsh tools/scripts/evd5-swap-evidence-routes.ps1
#   2>&1 | tee "${EVD9_ROOT}/forcing/route-swap-apply.txt"
# pwsh tools/scripts/evd5-swap-evidence-routes.ps1 -Check
#   2>&1 | tee "${EVD9_ROOT}/forcing/route-swap-postcheck.txt"

  4. Recreate only gateway and Console using their exact captured live config-file lists and reviewed image pins. Never run broad up -d against a reconstructed subset. Record commands before use.

  5. Verify first-match routing for the consolidated group, both Policy false friends, and the Platform /api/v2/evidence route before functional forcing.

9. Run the forcing functions

Each must cross the gateway and identify the successor build in response metadata or correlated logs. Record request ID, tenant, actor, result, persisted identifiers, digest, and timestamps; never record tokens or sensitive payloads.

Configuration precondition (EVD-13, added after the 2026-09-05 window). FF-1 and FF-3 below are only executable once the running render carries the consolidated host’s Attestor transparency and write keys. The 2026-09-05 window ran without them and both gates failed on configuration rather than on the data: assert them in the live render BEFORE running the forcing functions, not after interpreting a failure.

for role in stellaops-evidence-web stellaops-evidence-worker; do
  docker inspect "$role" --format '{{range .Config.Env}}{{println .}}{{end}}' \
    | grep -cE '^attestor__'          # expect 28
  docker exec "$role" test -f /app/etc/secrets/attestor-signing.pem && echo "$role KEY-MOUNTED"
done
docker inspect stellaops-evidence-web --format '{{range .Config.Env}}{{println .}}{{end}}' \
  | grep -E '^attestor__security__mtls__requireClientCertificate='   # must be false

If any is missing the host is running on product defaults that differ from what this estate ran: the write leg answers 403 attestor.validation.client_cert_required to every caller on every network (this host serves plain HTTP and never calls ConfigureAttestorKestrel, so no client certificate can ever be presented — true here is a kill switch, not a posture), and every entry that reaches the checkpoint stage fails checkpoint_public_key_missing. EvidenceConsolidationComposeConformanceTests pins the same key set in the COMMITTED overlay, so a render mismatch here means the running container is stale, not that the file drifted.

  1. Attestation (FF-1). Identity: a bearer client holding the canonical scope attest:create (the ASP.NET policy name is attestor:write). Path: POST /api/evidence/v1/rekor/entries through the gateway. Prove, in order: submission accepted with a rekor uuid; GET /api/evidence/v1/attestations/{uuid} 200 from the successor build; POST /api/evidence/v1/rekor/verify with that uuid returns ok:true; and the RFC 3161 leg via the unchanged TSA role (FF-2 exercises the same TSA and may be cited). Scope negative — read this before recording one. attestor:write is a RequireAssertion policy of “attest:create OR trusted-network service bypass”, and the gateway forwards from 172.19.0.0/16, which IS a bypass network. A no-scope negative run through the gateway therefore proves nothing about scope: it is admitted by address. Either run the negative from outside the bypass set, or record explicitly that the scope gate is not provable through the gateway (ROA-1 class, SPRINT_20260904_003). Do not record a gateway 403 from the certificate gate as a scope result — that was the 2026-09-05 window’s ambiguity.
  2. Evidence capsule: store an approved non-sensitive payload, record its SHA-256, retrieve it through /api/evidence/v1/evidence/**, and prove the returned bytes re-hash identically.
  3. Proof/Console (FF-3). Run this on a subject attested by FF-1’s own submission, not on a carried subject. Open it in deployed Console, render the chain, run “Verify all”, and require PROVEN. A direct API 200 without Console rendering does not satisfy this gate. Two Console behaviours decide whether you see anything at all: the custody list filter is an EXACT match on artifact sha256 / image digest / subject URI and Console sends the sha256:-prefixed digest, so only an imageDigest-bearing attestation is found; and only kinds in the timeline’s SCAN_KINDS / DEPLOY_KINDS / BUILD_KINDS / VERDICT_KINDS light a step — a report renders nothing. Expected residual: witness_missing (with policy_warn:transparency) stays on every entry and is a NON-blocking warn — Verification.RequireWitnessEndorsement defaults false and no witness aggregator is configured on this estate or was on the predecessor. It must not render as TAMPERED. Do NOT gate this on the carried entries, and do not reach for the SER-5b proof backfill to “fix” them. Measured 2026-09-05 with every key above in place (docs/implplan/_evidence/20260905-evd9-attestor-half-carry-over.md): all 26 carried entries verify ok:false BY DATA, on any configuration.
    • 23 rows are UIF-8 demo fixtures on log_backend='stellaops-transparency', a backend and log URL no writer in src/ produces; their checkpoint names a log that does not exist and their proofs carry unoriented hex hashes. POST /rekor/entries/proof-backfill?maxEntries=1000 returned 200 {"refreshed":31} and the md5 of those 23 rows’ proof JSON was byte-identical before and after — nothing re-derives a proof for a log that was never written. proof_leafhash_mismatch + proof_path_orientation_missing is the CORRECT verdict on them.
    • 3 rows are real local-log entries whose checkpoint notes were signed by the dev key KEY-2R retired (f11c76a5d8, 2026-08-21) because it had been committed. With the current key they move to checkpoint_signature_invalid. Making them pass would mean trusting a retired key. Record this as a data-provenance fact, not a configuration failure. Establish what wrote a row before concluding the host is misconfigured — that inversion cost the 2026-09-05 window a cycle.
  4. Repeat one protected endpoint anonymously. It must remain 401/403 and persist no row.

Unavailable or ambiguous ownership, stale timestamp, digest change, or Policy-owned path handled by Evidence triggers rollback.

10. Revoke predecessor access and soak

Only after parity and all forcing functions are green:

  1. Apply reviewed grants-forward.sql against the exact database named in its header. It revokes only recorded predecessor access; it does not drop a schema, role, or database.
  2. Prove the reviewed inverse remains available in grants-rollback.sql.
  3. Verify the Evidence owner remains NOSUPERUSER NOBYPASSRLS, all 13 EvidenceLocker RLS tables remain forced, both targets are running, and all Evidence doctor checks are healthy.
  4. Keep predecessor writers stopped but present. Retain their images, compose definitions, source schemas, and volumes through rollback retention.
  5. Soak for the approved duration. Capture target/gateway/Console/TSA/tileproxy errors, permission denials, queue backlog, retries, latency, and forcing identifiers. Any unexplained increase or predecessor write is an abort.

Close EVD-9 only after a signed soak receipt. EVD-10 and old-schema deletion remain separate work.

Rollback

Rollback is available because the source schemas are frozen and unchanged. Repoint back; do not restore over an intact source database.

  1. Declare the abort and keep the writer fence. Stop both targets.
  2. Restore the exact route, overlay, and Console blobs/image captured under rollback/; recreate only gateway and Console with their captured config-file lists. Verify the fifteen predecessor routes, wildcard-backed Console Attestor paths, both Policy false friends, and /api/v2/evidence before restarting writers.
  3. Apply reviewed grants-rollback.sql and verify predecessor access equals the pre-window receipt.
  4. Restart stellaops-attestor, stellaops-evidence-locker-web, and stellaops-evidence-locker-worker using captured image IDs and exact compose lists. Do not restart Signer, TSA, or tileproxy unless their own health requires it.
  5. Repeat predecessor health, authorization, capsule digest, attestation/timestamp, and Console proof checks. Confirm source counts still equal source-counts-b.tsv.
  6. Leave stellaops_evidence and its failed-window data intact for investigation unless the owner separately approves destruction. Do not drop the database or role as cleanup.

Restore source-domain-full.dump only if the source itself was unexpectedly modified or lost, and only through the separately approved migration recovery procedure. That is disaster recovery, not normal EVD-9 rollback.

Window closeout record

The signed record contains:

An absent receipt means the window remains open or rolls back; it never becomes a prose exception.