Authority — building and deploying the staged issuer fold (AUTH-8)
Sprint: SPRINT_20260722_016 AUTH-8 (consolidation recipe S8). Overlay: devops/compose/docker-compose.authority.yml. Companion runbook: docs/runbooks/authority/authority-cutover-auth5-route-swap.md (the route retarget; same window).
This family is the odd one out among the ten consolidation programs, and the difference decides how the whole window is shaped: issuer-directory folds into authority as a ROLE of it (owner ruling A8), so there is no new deployable, no new image, no new publish key and no new connection variable. The overlay only adds the fold-time environment to the authority key that already exists in docker-compose.stella-services.yml.
Everything below is repo-side. Nothing here has been applied to any running stack, and the AUTH-9 window is where it gets applied — under the owner’s window approval (MASTER §4 red line 1).
Canonical AUTH-9 window transaction
Superseded 2026-08-11 (AUTH-5). This section used to say the window could not open at all, because nothing mapped the folded endpoints. That is fixed: the Authority host now maps them from StellaOps.IssuerDirectory.Api — the library AUTH-4 lifted, so no deployable is referenced (P19) — behind ISSUERDIRECTORY__FOLD__ENABLED, which defaults to false. D-AUTH7-5 is closed and the work landed on AUTH-5, which owns S5.
The route, the direct callers and the stopped container are one maintenance transaction. The write fence stays in place from the first parity count through the end of the soak (or through rollback). Half-applications fail differently:
| Applied | Not applied | Symptom |
|---|---|---|
| gateway retarget | fold switch | 404 farm with a GREEN route suite (the AAI-8 note-2 failure mode). The route tests assert the route table, not that anyone serves the path. |
| fold switch | direct-consumer repoint | Platform and the two Excititor roles keep writing to the legacy host even after the gateway is correct. Split brain. |
| fold switch | container stop | two services can answer /issuer-directory; which one receives a call depends on the caller’s base address. Split brain. |
| container stop | gateway retarget | the prefix 502/503s while the retarget is still pointing at the stopped host. |
There is one canonical order. The retirement section and the AUTH-5 companion runbook refer back to this list; they do not define a second order:
- Establish the maintenance write fence. Record the running image/config-files labels, database snapshot, source/target counts, and all six compiled-consumer dispositions plus Platform’s non-SDK direct HTTP caller. The complete measured inventory is below; an unexplained caller is a NO-GO.
- Apply the exact
authority_adminowner/RLS procedure below and change only the protectedSTELLAOPS_POSTGRES_AUTHORITY_CONNECTIONvalue to use that role. Render every service’s exact current compose chain before recreating anything. - AUTH-9 copies the
issuerdata — measured on this estate 2026-08-11 as a zero-row copy; the source tables instellaops_platformare all empty, so the parity gate is 0 = 0. Re-measure rather than inherit. - Bring
authorityup with the overlay ANDAUTHORITY_ISSUER_DIRECTORY_FOLD_ENABLED=true. Verify the surface answers on the container before touching the gateway: an unauthenticatedGET /issuer-directory/issuersagainst the authority container must be 401, and a token withoutissuer-directory:readmust be 403. A 404 here means the switch did not take — stop, do not retarget. - In the ignored/private env file used by each recorded chain, set
AUTHORITY_ISSUER_DIRECTORY_DIRECT_BASE_ADDRESS=http://authority.stella-ops.local. Recreate the current measured set —platform,excititor-web, andexcititor-worker— each against its own recorded compose chain. If a recorded chain includes the opt-in operator-signing proof overlay, also recreatepolicy-engineandrelease-orchestrator. Then proveSTELLAOPS_ISSUERDIRECTORY_URL/IssuerDirectory__Client__BaseAddressrender to Authority and no longer nameissuerdirectory.stella-ops.local. Do not borrow another service’s-flist. - Run
pwsh tools/scripts/auth5-swap-issuer-directory-routes.ps1 -Check, then apply the AUTH-5 retarget. Through the gateway, prove the unauthorised pair and create/read/delete the deterministic no-key issuer canary described below. The delete must finish before the stop; no active canary row may survive. - Immediately stop the legacy container by NAME:
docker stop stellaops-issuer-directory. Confirm that the gateway and every direct caller still reach Authority. This is the atomic route/stop boundary; do not put a long-running check between steps 6 and 7. - Run the login and issuer-trust evaluation, then the owner-approved
AUTH9-TENANT-DISPLAY-NAME-REVERSIBLE-V1tenant forcing pair in section 9 and the permission-error soak. The pair PATCHes one owner-designated disposable tenant, proves Authority persistence/audit plus Policy’s named local replica/checkpoint, and immediately PATCHes the exact prior display name back. It never creates or deletes a tenant and never uses SQL to mutate tenant data.
Reviewed pre-window execution sheet (AUTH-9)
This is the one execution sheet for the window. It fills the four omissions found by the 2026-08-22 no-go: an enforceable source-writer fence, an identified and restore-proven snapshot, a current clean Authority image whose pin wins the recorded compose chain, and a recreate list derived from containers that are actually running. Commands are for Git Bash on the Docker host, except the explicit pwsh route command. Run from the repository root. Nothing in this section was run merely by committing the runbook.
0. Fill the reviewed record; placeholders are not executable values
Keep the record outside Git because it names a backup path and a protected env file. It must contain no token, password, rendered compose JSON, or connection string.
set -euo pipefail
AUTH9_WINDOW_ID='<REQUIRED approved window/change id>'
AUTH9_APPROVED_SOURCE_SHA='<REQUIRED full 40-character reviewed commit>'
AUTH9_ACTIONS_SOURCE_SHA='<REQUIRED commit covered by the green Actions run>'
AUTH9_ACTIONS_RUN_URL='<REQUIRED green forge run URL, or a substitute-run receipt path>'
AUTH9_ENV_FILE='<REQUIRED absolute path to the protected compose env file>'
AUTH9_BACKUP_ROOT='<REQUIRED absolute path on restore-capable storage>'
AUTH9_SOAK_MINUTES='<REQUIRED approved integer duration>'
AUTH9_AUTH6_RUN_ID='<REQUIRED unused run-NNN evidence id>'
AUTH9_EVIDENCE_DATE='<REQUIRED UTC date YYYY-MM-DD>'
AUTH9_TENANT_FORCING_PROCEDURE_ID='AUTH9-TENANT-DISPLAY-NAME-REVERSIBLE-V1'
AUTH9_DISPOSABLE_TENANT='<REQUIRED owner-designated existing disposable tenant slug>'
AUTH9_DISPOSABLE_TENANT_APPROVAL='<REQUIRED approval/change record naming that tenant disposable>'
auth9_require_actual() {
local name="$1" value="${!1-}"
case "$value" in
''|*'<'*|*'>'*|*REQUIRED*|*TBD*|*TODO*)
echo "NO-GO: $name is not an actual reviewed value" >&2
return 1 ;;
esac
}
# The verification evidence is EITHER a forge run URL OR a local substitute-run
# receipt, and either way it must EXIST. `auth9_require_actual` alone would pass
# a receipt path that was never written, which is the one failure this gate
# cannot afford: it would let the window proceed on evidence nobody produced.
#
# The substitute exists because on 2026-09-07 the forge could not supply a green
# run for ANY commit (test-architecture.yml 0 of 121 runs, compose-healthchecks
# 0 of 120, a 157-run backlog) -- a CI health defect owned by
# SPRINT_20260825_001, not an AUTH-9 condition. The substitute runs the SAME
# commands those workflows run, locally, at the exact commit the image is built
# from. A receipt records which commands ran and states the FORGE side as
# UNVERIFIED. It must never be written up as "green": the local run is the
# evidence and the forge is ABSENT, and a receipt that launders absence into a
# tick is worse than no receipt at all.
auth9_require_actions_evidence() {
local value="${AUTH9_ACTIONS_RUN_URL-}"
case "$value" in
https://*) return 0 ;;
esac
if [[ -f "$value" && -s "$value" ]]; then
grep -Fq 'forge: UNVERIFIED' "$value" || {
echo "NO-GO: substitute receipt $value does not state the forge side as UNVERIFIED" >&2
return 1
}
return 0
fi
echo "NO-GO: AUTH9_ACTIONS_RUN_URL is neither an https URL nor an existing non-empty receipt file: $value" >&2
return 1
}
for name in AUTH9_WINDOW_ID AUTH9_APPROVED_SOURCE_SHA AUTH9_ACTIONS_SOURCE_SHA \
AUTH9_ACTIONS_RUN_URL AUTH9_ENV_FILE AUTH9_BACKUP_ROOT AUTH9_SOAK_MINUTES \
AUTH9_AUTH6_RUN_ID AUTH9_EVIDENCE_DATE \
AUTH9_TENANT_FORCING_PROCEDURE_ID AUTH9_DISPOSABLE_TENANT \
AUTH9_DISPOSABLE_TENANT_APPROVAL; do
auth9_require_actual "$name"
done
auth9_require_actions_evidence
[[ "$AUTH9_APPROVED_SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$AUTH9_ACTIONS_SOURCE_SHA" == "$AUTH9_APPROVED_SOURCE_SHA" ]]
[[ "$AUTH9_SOAK_MINUTES" =~ ^[1-9][0-9]*$ ]]
[[ "$AUTH9_AUTH6_RUN_ID" =~ ^run-[0-9]{3}$ ]]
[[ "$AUTH9_EVIDENCE_DATE" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]
[[ "$AUTH9_TENANT_FORCING_PROCEDURE_ID" == 'AUTH9-TENANT-DISPLAY-NAME-REVERSIBLE-V1' ]]
[[ "$AUTH9_DISPOSABLE_TENANT" =~ ^[a-z0-9][a-z0-9-]*$ ]]
[[ "$AUTH9_DISPOSABLE_TENANT" != default ]]
test -s "$AUTH9_ENV_FILE"
test -d "$AUTH9_BACKUP_ROOT"
command -v docker >/dev/null
command -v jq >/dev/null
command -v sha256sum >/dev/null
AUTH9_WORK_DIR="tmp/auth9-${AUTH9_WINDOW_ID}"
# The guard is on section 0's OWN marker, not on the directory. The pre-window
# substitute run (D-AUTH9-18) legitimately writes actions-substitute/ into this
# directory before section 0 executes, and a bare `test ! -e` on the directory
# would refuse the very evidence AUTH9_ACTIONS_RUN_URL points at. Keyed on the
# marker, a second section-0 run against the same window id is still refused,
# which is what the guard was protecting.
test ! -e "$AUTH9_WORK_DIR/reviewed-inputs.txt" || {
echo "NO-GO: section 0 already ran for this window: $AUTH9_WORK_DIR/reviewed-inputs.txt" >&2
exit 1
}
mkdir -p "$AUTH9_WORK_DIR"
printf '%s\n' \
"window=$AUTH9_WINDOW_ID" \
"source_sha=$AUTH9_APPROVED_SOURCE_SHA" \
"actions_source_sha=$AUTH9_ACTIONS_SOURCE_SHA" \
"actions_run=$AUTH9_ACTIONS_RUN_URL" \
"soak_minutes=$AUTH9_SOAK_MINUTES" \
"auth6_run_id=$AUTH9_AUTH6_RUN_ID" \
"evidence_date=$AUTH9_EVIDENCE_DATE" \
> "$AUTH9_WORK_DIR/reviewed-inputs.txt"
The disposable tenant and its approval record are reviewed inputs, not values the operator may invent in the window. default is structurally refused. The procedure also refuses a missing/non-active tenant or a Policy replica that is not already converged before the first mutation. Divergent name and display_name values are valid: the PATCH contract changes canonical name only and preserves display_name byte-for-byte.
1. Build current source from an isolated clean worktree and prepare the final pin
Do this before the outage. A green run for a different commit is a NO-GO. The image is accepted only when its baked build information proves exact source and a clean tree.
git cat-file -e "${AUTH9_APPROVED_SOURCE_SHA}^{commit}"
AUTH9_BUILD_WORKTREE="$AUTH9_WORK_DIR/build-worktree"
git worktree add --detach "$AUTH9_BUILD_WORKTREE" "$AUTH9_APPROVED_SOURCE_SHA"
test -z "$(git -C "$AUTH9_BUILD_WORKTREE" status --porcelain)"
AUTH9_IMAGE_TAG="auth9-${AUTH9_APPROVED_SOURCE_SHA:0:12}"
(
cd "$AUTH9_BUILD_WORKTREE"
IMAGE_TAG="$AUTH9_IMAGE_TAG" bash devops/docker/build-service-publish.sh authority
)
AUTH9_IMAGE_REF="stellaops/authority:${AUTH9_IMAGE_TAG}"
AUTH9_IMAGE_ID="$(docker image inspect "$AUTH9_IMAGE_REF" --format '{{.Id}}')"
[[ "$AUTH9_IMAGE_ID" =~ ^sha256:[0-9a-f]{64}$ ]]
MSYS_NO_PATHCONV=1 docker run --rm --entrypoint sh "$AUTH9_IMAGE_REF" \
-lc 'cat /app/buildinfo.json' > "$AUTH9_WORK_DIR/authority-buildinfo.json"
jq -e --arg sha "$AUTH9_APPROVED_SOURCE_SHA" \
'.gitSha == $sha and .worktreeState == "clean" and .dirtyFileCount == 0' \
< "$AUTH9_WORK_DIR/authority-buildinfo.json" >/dev/null
AUTH9_FINAL_PIN="$AUTH9_WORK_DIR/docker-compose.authority-auth9-final-pin.yml"
cat > "$AUTH9_FINAL_PIN" <<YAML
services:
authority:
image: ${AUTH9_IMAGE_ID}
pull_policy: never
environment:
Doctor__Registration__PlatformBaseAddress: http://platform.stella-ops.local
Doctor__Registration__SelfEndpoint: http://authority.stella-ops.local/doctor/authority/checks
Doctor__Registration__Tenant: default
YAML
printf '%s\n' "image_ref=$AUTH9_IMAGE_REF" "image_id=$AUTH9_IMAGE_ID" \
> "$AUTH9_WORK_DIR/authority-image-record.txt"
Do not overwrite or delete the inherited pin file. The window appends $AUTH9_FINAL_PIN after every label-recorded file and after the fold overlay. It names the immutable local image ID with pull_policy: never, never the mutable build tag. The render and post-recreate image-ID checks in section 7 prove last-wins rather than assuming it.
2. Record the exact running caller and recreate inventory
The label set is the authority for each container’s project, service key, working directory and ordered config chain. Missing labels, a missing chain file, or a running caller outside the reviewed set is a NO-GO. ABSENT is a valid recorded disposition for either Excititor role and is not an instruction to create it.
: > "$AUTH9_WORK_DIR/container-inventory.tsv"
auth9_record_container() {
local container="$1" required="$2"
if ! docker inspect "$container" >/dev/null 2>&1; then
printf '%s\tABSENT\n' "$container" >> "$AUTH9_WORK_DIR/container-inventory.tsv"
[[ "$required" == optional ]] || {
echo "NO-GO: required container absent: $container" >&2; return 1;
}
return 0
fi
local state project service working chain image_id chain_file path
state="$(docker inspect "$container" --format '{{.State.Status}}')"
project="$(docker inspect "$container" --format '{{index .Config.Labels "com.docker.compose.project"}}')"
service="$(docker inspect "$container" --format '{{index .Config.Labels "com.docker.compose.service"}}')"
working="$(docker inspect "$container" --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}')"
chain="$(docker inspect "$container" --format '{{index .Config.Labels "com.docker.compose.project.config_files"}}')"
image_id="$(docker inspect "$container" --format '{{.Image}}')"
[[ "$state" == running && -n "$project" && -n "$service" && -n "$working" && -n "$chain" ]] || {
echo "NO-GO: $container is not running or lacks compose reconstruction labels" >&2; return 1;
}
chain_file="$AUTH9_WORK_DIR/${container}.config-files"
: > "$chain_file"
IFS=',' read -r -a files <<< "$chain"
# Absolute-path test as a REGEX IN A VARIABLE, not a case glob. A bracket
# expression containing a backslash does not match a literal backslash in the
# MSYS bash this estate's Docker host runs (verified 5.2.37 on 2026-09-07): the
# old `case "$path" in /*|[A-Za-z]:[\\/]*)` matched C:/dev/x.yml but NOT
# C:\dev\x.yml, and every compose label on this host uses backslashes. Every
# already-absolute chain entry was therefore reclassified as relative, had the
# working directory prefixed onto it, and failed the `test -f` below, aborting
# the window at section 2 with a false "missing compose file" NO-GO on all 33
# files. Do not fold this back into a glob.
auth9_abs_path_re='^([A-Za-z]:[\\/]|/)'
for path in "${files[@]}"; do
[[ "$path" =~ $auth9_abs_path_re ]] || path="$working/$path"
test -f "$path" || {
echo "NO-GO: $container label names missing compose file: $path" >&2; return 1;
}
printf '%s\n' "$path" >> "$chain_file"
done
printf '%s\tRUNNING\t%s\t%s\t%s\t%s\n' \
"$container" "$project" "$service" "$image_id" "$working" \
>> "$AUTH9_WORK_DIR/container-inventory.tsv"
}
auth9_record_container stellaops-router-gateway required
auth9_record_container stellaops-authority required
auth9_record_container stellaops-issuer-directory required
auth9_record_container stellaops-platform required
auth9_record_container stellaops-excititor-web optional
auth9_record_container stellaops-excititor-worker optional
auth9_record_container stellaops-policy-engine optional
auth9_record_container stellaops-release-orchestrator optional
: > "$AUTH9_WORK_DIR/running-callers.txt"
while IFS= read -r container; do
if docker inspect "$container" --format '{{range .Config.Env}}{{println .}}{{end}}' \
| grep -Eq '^(STELLAOPS_ISSUERDIRECTORY_URL|IssuerDirectory__Client__BaseAddress)='; then
case "$container" in
stellaops-platform|stellaops-excititor-web|stellaops-excititor-worker|\
stellaops-policy-engine|stellaops-release-orchestrator) ;;
*) echo "NO-GO: unreviewed running issuer caller: $container" >&2; exit 1 ;;
esac
printf '%s\n' "$container" >> "$AUTH9_WORK_DIR/running-callers.txt"
fi
done < <(docker ps --format '{{.Names}}')
sort -u -o "$AUTH9_WORK_DIR/running-callers.txt" "$AUTH9_WORK_DIR/running-callers.txt"
grep -qx 'stellaops-platform' "$AUTH9_WORK_DIR/running-callers.txt"
for container in stellaops-policy-engine stellaops-release-orchestrator; do
if grep -qx "$container" "$AUTH9_WORK_DIR/running-callers.txt"; then
grep -q 'docker-compose.operator-signing-liveproof.override.yml' \
"$AUTH9_WORK_DIR/${container}.config-files" || {
echo "NO-GO: $container is an active caller without the reviewed opt-in overlay" >&2
exit 1
}
fi
done
printf '%s\t%s\n' \
'StellaOps.Excititor.Attestation' 'LIBRARY_INHERITS_EXCITITOR_HOST' \
'StellaOps.DeltaVerdict' 'LIBRARY_NO_PRODUCTION_REGISTRATION' \
>> "$AUTH9_WORK_DIR/container-inventory.tsv"
The source conformance guard pins the six SDK-project dispositions. This runtime sweep adds the seventh non-SDK caller (Platform), detects optional callers by their actual environment, and refuses an eighth caller.
3. Establish and prove the maintenance writer fence
First stop external ingress, every discovered direct caller, and the target Authority process. Leave the legacy issuer container running until the canonical route/canary boundary, but lock its four source tables in PostgreSQL SHARE mode. SHARE permits the snapshot/parity reads and conflicts with ROW EXCLUSIVE, so inserts/updates/deletes cannot cross the fence even if an unreviewed source writer exists. The lock holder is named and mechanically verified; a four-hour sleep is only a dead-man timeout, not the release command.
mapfile -t AUTH9_RUNNING_CALLERS < "$AUTH9_WORK_DIR/running-callers.txt"
AUTH9_STOP_SET=(stellaops-router-gateway "${AUTH9_RUNNING_CALLERS[@]}" stellaops-authority)
printf '%s\n' "${AUTH9_STOP_SET[@]}" | awk '!seen[$0]++' \
> "$AUTH9_WORK_DIR/writer-fence-stopped.txt"
while IFS= read -r container; do docker stop "$container" >/dev/null; done \
< "$AUTH9_WORK_DIR/writer-fence-stopped.txt"
while IFS= read -r container; do
test -z "$(docker ps --filter "name=^/${container}$" --format '{{.Names}}')" || {
echo "NO-GO: writer-fence container still running: $container" >&2; exit 1;
}
done < "$AUTH9_WORK_DIR/writer-fence-stopped.txt"
http_code="$(curl -sk --connect-timeout 3 --max-time 5 -o /dev/null \
-w '%{http_code}' https://stella-ops.local/issuer-directory/issuers || true)"
[[ "$http_code" == 000 ]] || {
echo "NO-GO: front door remains reachable during hard fence (HTTP $http_code)" >&2; exit 1;
}
AUTH9_SOURCE_FENCE_APP="auth9.source-fence.${AUTH9_WINDOW_ID}"
docker exec -d \
-e PGAPPNAME="$AUTH9_SOURCE_FENCE_APP" \
-e PGOPTIONS='-c lock_timeout=10000 -c statement_timeout=0' \
stellaops-postgres psql -X -v ON_ERROR_STOP=1 -U stellaops \
-d stellaops_platform -c \
'BEGIN; LOCK TABLE issuer.issuers, issuer.issuer_keys, issuer.trust_overrides, issuer.audit IN SHARE MODE; SELECT pg_sleep(14400); ROLLBACK;'
# NO psql `-v name=value` / `:'name'` ANYWHERE IN THIS RUNBOOK, and the reason is
# measured rather than stylistic. On this host psql does NOT interpolate a
# variable inside `-c`: it passes the literal `:'name'` to the server, which
# answers `ERROR: syntax error at or near ":"`. Measured three ways on 2026-09-08
# (window 3, D-AUTH9-35) -- `-v` before `-d`, after `-d`, and with and without
# MSYS_NO_PATHCONV -- all three fail identically.
#
# It fails in the direction that stops a window on a healthy estate: the gate
# below reads an EMPTY count and refuses. That is exactly what it did. The fence
# was perfect, all four ShareLocks were granted the whole time, and the window
# lost about ninety seconds inside its outage to a false NO-GO before the same
# count was re-read inlined and returned 4.
#
# Every value this runbook binds is already a reviewed, shape-validated constant
# -- the window id, the tenant slug (`^[a-z0-9][a-z0-9-]*$`, refused when it is
# `default`), generated hex correlation ids -- so they are INLINED into the SQL
# and the validation stays at the point the value is admitted, in section 0.
# Before reintroducing `-v`, run `psql -v t=x -c "SELECT :'t';"` on the host and
# read the result.
#
# The count MUST come from stellaops_platform, the database that OWNS the locked
# relations, not from `postgres`. pg_locks and pg_stat_activity are cluster-wide, but
# pg_class and pg_namespace are PER-DATABASE: joined from `postgres` the relation OIDs
# resolve to nothing, the inner join yields no rows, and this gate reports 0 while the
# fence is perfect - a fail-closed NO-GO on a correct result. Measured 2026-09-07 in the
# executed window (D-AUTH9-34): the holder's four locks were granted the whole time, with
# a NULL relname and database=stellaops_platform when read from `postgres`.
for _ in $(seq 1 20); do
lock_count="$(docker exec stellaops-postgres psql -XqAt -U stellaops -d stellaops_platform -c \
"SELECT count(*) FROM pg_locks l JOIN pg_class c ON c.oid=l.relation JOIN pg_namespace n ON n.oid=c.relnamespace JOIN pg_stat_activity a ON a.pid=l.pid WHERE a.application_name='$AUTH9_SOURCE_FENCE_APP' AND n.nspname='issuer' AND c.relname IN ('issuers','issuer_keys','trust_overrides','audit') AND l.mode='ShareLock' AND l.granted;")"
[[ "$lock_count" == 4 ]] && break
sleep 1
done
[[ "${lock_count:-0}" == 4 ]] || {
echo 'NO-GO: source fence did not acquire four granted ShareLock rows' >&2; exit 1;
}
target_sessions="$(docker exec stellaops-postgres psql -XqAt -U stellaops -d postgres -c \
"SELECT count(*) FROM pg_stat_activity WHERE datname='stellaops_authority';")"
[[ "$target_sessions" == 0 ]] || {
echo "NO-GO: target database still has $target_sessions session(s)" >&2; exit 1;
}
AUTH9_WRITER_FENCE_STATE=VERIFIED
printf '%s\n' "writer_fence=$AUTH9_WRITER_FENCE_STATE" \
"source_fence_application=$AUTH9_SOURCE_FENCE_APP" \
> "$AUTH9_WORK_DIR/writer-fence-proof.txt"
If any command after this point fails, keep the source lock, do not start an unrecorded caller, and enter section 10. Release the source lock only with the explicit command in section 9 or after rollback has fully restored the legacy path.
4. Take, identify, verify, and scratch-restore a fresh snapshot
Amended 2026-09-07 (D-AUTH9-11): the gate is a PER-DATABASE custom-format pg_dump of the two databases this window touches, not a cluster pg_basebackup. The cluster form is retained as a note at the end for hosts with the headroom; on the estate this runbook is written for it is not merely slow, it is destructive.
Measured on the Docker host 2026-09-07: /var/lib/postgresql/data is 215 GB (stellaops_vuln 99 GB, stellaops_findings 96 GB) while C: had 44 GB free at 96 % used, falling to 38 GB within the same session as peer lanes built. A cluster backup plus its scratch restore is roughly 430 GB of writes against 40 GB of headroom. It does not fail cleanly: it fills C:, which is the head of the documented disk to VHDX to postgres cascade (postgres dies with 53100: No space left on device, stellaops-platform crash-loops at StartupMigrationHost, gateway and authority go unhealthy). Taking a backup that takes the estate down is not a safety measure. Check the headroom before choosing a form, and do not inherit either verdict:
df -h /c | tail -1
docker exec stellaops-postgres sh -lc 'du -sh /var/lib/postgresql/data | tail -1'
Two databases matter here and both are small in logical form: the SOURCE is schema issuer in stellaops_platform, and the TARGET is the whole of stellaops_authority (721 GB-reported but 31 MB as a custom-format dump — the difference is index and bloat, which a logical dump does not carry).
Two defects a filed version of this command carried, both found by the EVD-10b lane and both fixed here: restore as -U stellaops, and use no --schema filter on the restore. A schema filter on pg_restore silently restores nothing while exiting 0, which turns the whole proof into a green-that-cannot-go-red.
MSYS_NO_PATHCONV=1 is required on every docker exec naming a container-absolute path; see the note in section 6.
auth9_issuer_counts() {
local database="$1"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres psql -XqAt -F '|' -U stellaops -d "$database" -c \
"SELECT current_database(), table_name, row_count FROM (
SELECT 'audit' AS table_name, count(*) AS row_count FROM issuer.audit
UNION ALL SELECT 'issuer_keys', count(*) FROM issuer.issuer_keys
UNION ALL SELECT 'issuers', count(*) FROM issuer.issuers
UNION ALL SELECT 'trust_overrides', count(*) FROM issuer.trust_overrides
) AS counts ORDER BY table_name;"
}
{
auth9_issuer_counts stellaops_platform
auth9_issuer_counts stellaops_authority
} > "$AUTH9_WORK_DIR/pre-snapshot-issuer-counts.tsv"
# Every-table counts, used as the restore comparison. Live-write tables are
# expected to drift against a frozen dump; take these INSIDE the writer fence
# (section 3) and the drift is zero. Outside the fence, expect
# `authority.login_attempts` to differ by roughly one row per second.
auth9_all_table_counts() {
local database="$1"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres psql -XqAt -U stellaops -d "$database" -c \
"SELECT n.nspname||'.'||c.relname||'='||
(xpath('/row/c/text()', query_to_xml(
format('SELECT count(*) AS c FROM %I.%I', n.nspname, c.relname),
false, true, '')))[1]::text::bigint
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind = 'r' AND n.nspname NOT IN ('pg_catalog','information_schema')
ORDER BY 1;"
}
AUTH9_DUMP_DIR="/tmp/auth9-${AUTH9_WINDOW_ID}-snapshot"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres mkdir -p "$AUTH9_DUMP_DIR"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres pg_dump -U stellaops \
-d stellaops_platform -Fc -n issuer -f "$AUTH9_DUMP_DIR/issuer-source.dump"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres pg_dump -U stellaops \
-d stellaops_authority -Fc -f "$AUTH9_DUMP_DIR/authority-full.dump"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres sh -lc \
"cd '$AUTH9_DUMP_DIR' && sha256sum authority-full.dump issuer-source.dump" \
| tee "$AUTH9_WORK_DIR/dump-hashes.txt"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres pg_restore --list \
"$AUTH9_DUMP_DIR/issuer-source.dump" > "$AUTH9_WORK_DIR/issuer-source.toc.txt"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres pg_restore --list \
"$AUTH9_DUMP_DIR/authority-full.dump" > "$AUTH9_WORK_DIR/authority-full.toc.txt"
test -s "$AUTH9_WORK_DIR/issuer-source.toc.txt"
test -s "$AUTH9_WORK_DIR/authority-full.toc.txt"
# Scratch restore. The names are fixed and asserted before the drop, so this
# can never drop a real database.
AUTH9_PROBE_TARGET="auth9_probe_${AUTH9_WINDOW_ID//[^a-z0-9]/_}_authority"
AUTH9_PROBE_SOURCE="auth9_probe_${AUTH9_WINDOW_ID//[^a-z0-9]/_}_source"
for db in "$AUTH9_PROBE_TARGET" "$AUTH9_PROBE_SOURCE"; do
[[ "$db" == auth9_probe_* ]] || { echo "NO-GO: probe name is not auth9_probe_*: $db" >&2; exit 1; }
[[ "$(MSYS_NO_PATHCONV=1 docker exec stellaops-postgres psql -XqAt -U stellaops -d postgres \
-c "SELECT count(*) FROM pg_database WHERE datname='$db';")" == 0 ]] || {
echo "NO-GO: scratch database already exists: $db" >&2; exit 1; }
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres createdb -U stellaops "$db"
done
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres pg_restore -U stellaops \
-d "$AUTH9_PROBE_TARGET" --no-owner --no-privileges --exit-on-error \
"$AUTH9_DUMP_DIR/authority-full.dump"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres pg_restore -U stellaops \
-d "$AUTH9_PROBE_SOURCE" --no-owner --no-privileges --exit-on-error \
"$AUTH9_DUMP_DIR/issuer-source.dump"
auth9_all_table_counts stellaops_authority > "$AUTH9_WORK_DIR/counts-live-authority.txt"
auth9_all_table_counts "$AUTH9_PROBE_TARGET" > "$AUTH9_WORK_DIR/counts-restored-authority.txt"
auth9_all_table_counts stellaops_platform | grep '^issuer\.' \
> "$AUTH9_WORK_DIR/counts-live-source.txt"
auth9_all_table_counts "$AUTH9_PROBE_SOURCE" > "$AUTH9_WORK_DIR/counts-restored-source.txt"
diff -u "$AUTH9_WORK_DIR/counts-live-source.txt" "$AUTH9_WORK_DIR/counts-restored-source.txt"
diff -u "$AUTH9_WORK_DIR/counts-live-authority.txt" \
"$AUTH9_WORK_DIR/counts-restored-authority.txt"
for db in "$AUTH9_PROBE_TARGET" "$AUTH9_PROBE_SOURCE"; do
[[ "$db" == auth9_probe_* ]] || { echo "NO-GO: refusing to drop $db" >&2; exit 1; }
case "$db" in stellaops_*|postgres|template*) echo "NO-GO: protected name $db" >&2; exit 1 ;; esac
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres dropdb -U stellaops "$db"
done
[[ "$(MSYS_NO_PATHCONV=1 docker exec stellaops-postgres psql -XqAt -U stellaops -d postgres \
-c "SELECT count(*) FROM pg_database WHERE datname LIKE 'auth9_probe_%';")" == 0 ]]
AUTH9_SNAPSHOT_ID="dump/${AUTH9_WINDOW_ID}/$(awk '{print $1}' "$AUTH9_WORK_DIR/dump-hashes.txt" | tr '\n' '/' )"
AUTH9_RESTORE_PROOF_ID="restore/${AUTH9_WINDOW_ID}/counts-equal"
printf '%s\n' \
"form=per-database-custom-format-pg_dump" \
"reason=cluster basebackup impossible on this host (D-AUTH9-11)" \
"dump_dir_in_container=$AUTH9_DUMP_DIR" \
"snapshot_id=$AUTH9_SNAPSHOT_ID" \
"restore_proof_id=$AUTH9_RESTORE_PROOF_ID" \
> "$AUTH9_WORK_DIR/snapshot-restore-record.txt"
for name in AUTH9_SNAPSHOT_ID AUTH9_RESTORE_PROOF_ID; do
auth9_require_actual "$name"
done
Keep the dumps until the window closes and the rollback-retention period ends. They are the only recovery point this form provides, and unlike the cluster form they restore ONE database at a time, which is what a routine AUTH-9 rollback would ever want.
Note, for a host that does have the headroom. The original cluster form was backup-postgres.sh --yes --label auth9-<id> --out-dir <root>, accepted only when exactly one fresh postgres-basebackup-* artifact existed, its verify-backup.sh passed, and its sidecar supplied label, PostgreSQL system identifier, stop LSN and SHA-256; the proof was a --network none scratch restore reproducing the system identifier and both count sets. It gives Platform and Authority one shared recovery point, which the per-database form does not. Use it only where df shows free space greater than twice the data directory, and note that even there a cluster restore is disaster recovery, never the routine rollback — that remains the owner-role reversal in section 10.
5. Apply the owner procedure and protected connection switch
Run Exact database owner/grant/RLS procedure sections 1–3 below verbatim. Update only STELLAOPS_POSTGRES_AUTHORITY_CONNECTION in $AUTH9_ENV_FILE, using the secret-safe input described there. Before continuing, all four positive fields, both negative controls, and zero generic-owned objects must pass. Do not render the connection string into the evidence directory.
6. Re-measure, copy only when needed, and prove byte parity
The deterministic fingerprint is SHA-256 over ordered JSON rows. The target-only operator_provider_change_receipts table is deliberately absent. Identical fingerprint files mean no copy; otherwise every target DOMAIN table must be empty before the custom-format four-table data restore.
Parity is asserted on the three DOMAIN tables only, and issuer.audit is recorded as evidence rather than gated (amended 2026-09-08, window-4 pre-flight). issuer.audit is append-only history that the TARGET legitimately accumulates: the first fold window to serve a mutation writes audit rows into stellaops_authority, and AUTH-9’s own task text keeps them - “Target-only issuer audit rows and Authority audit/outbox history are retained as evidence”. Measured read-only before window 4: source 0/0/0 domain and 0 audit; target 0/0/0 domain and 4 audit - window 3’s forcing pair at 12:40:00-12:40:02Z, which is AUTH-21’s and AUTH-23’s retained live proof and must not be deleted. With audit inside the fingerprint, cmp no longer matched, the emptiness guard then refused with “target differs and is not empty”, and the post-copy diff would have failed for the same reason - so the gate would have NO-GO’d inside the outage on evidence the window itself was told to keep. Windows 1-3 read NOOP_ALREADY_EQUAL only because no window had yet written to the target. The emptiness precondition exists to stop two POPULATED issuer directories being merged, which is a property of the domain tables; audit counts are recorded before and after on both sides instead.
auth9_issuer_fingerprints() {
local database="$1" output="$2" table
: > "$output"
for table in issuers issuer_keys trust_overrides; do
count="$(docker exec stellaops-postgres psql -XqAt -U stellaops -d "$database" \
-c "SELECT count(*) FROM issuer.${table};")"
digest="$(docker exec stellaops-postgres psql -XqAt -U stellaops -d "$database" \
-c "COPY (SELECT row_to_json(t)::text FROM issuer.${table} AS t ORDER BY id) TO STDOUT" \
| sha256sum | cut -d' ' -f1)"
printf '%s\t%s\t%s\n' "$table" "$count" "$digest" >> "$output"
done
}
# Evidence, never a gate. Both sides, before and after, so a reviewer can see
# exactly what the target already held and what the copy added.
auth9_issuer_audit_counts() {
local database="$1" output="$2"
docker exec stellaops-postgres psql -XqAt -U stellaops -d "$database" \
-c "SELECT count(*) FROM issuer.audit;" | tr -d ' \r' > "$output"
}
auth9_issuer_fingerprints stellaops_platform "$AUTH9_WORK_DIR/issuer-source.before.tsv"
auth9_issuer_fingerprints stellaops_authority "$AUTH9_WORK_DIR/issuer-target.before.tsv"
auth9_issuer_audit_counts stellaops_platform "$AUTH9_WORK_DIR/issuer-source-audit.before.txt"
auth9_issuer_audit_counts stellaops_authority "$AUTH9_WORK_DIR/issuer-target-audit.before.txt"
if cmp -s "$AUTH9_WORK_DIR/issuer-source.before.tsv" "$AUTH9_WORK_DIR/issuer-target.before.tsv"; then
AUTH9_COPY_DISPOSITION=NOOP_ALREADY_EQUAL
else
awk -F '\t' '$2 != 0 { exit 1 }' "$AUTH9_WORK_DIR/issuer-target.before.tsv" || {
echo 'NO-GO: a target DOMAIN table is not empty; do not merge two authorities' >&2; exit 1;
}
AUTH9_COPY_DISPOSITION=COPY_SOURCE_TO_EMPTY_TARGET
AUTH9_DUMP_IN_CONTAINER="/tmp/auth9-${AUTH9_WINDOW_ID}-issuer.dump"
# MSYS_NO_PATHCONV=1 is REQUIRED on all three calls below, not decoration. The
# path is absolute INSIDE the container; Git Bash on the Docker host rewrites a
# leading-slash argument into a Windows path before docker sees it, so pg_dump
# receives a host path that does not exist and fails with `could not open output
# file ".../auth9-...-issuer.dump": No such file or directory` (reproduced
# 2026-09-07). Section 1 already sets it on its `docker run`; these were missed.
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres pg_dump -U stellaops -d stellaops_platform \
-Fc --data-only --no-owner --no-privileges \
-t issuer.issuers -t issuer.issuer_keys -t issuer.trust_overrides -t issuer.audit \
-f "$AUTH9_DUMP_IN_CONTAINER"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres pg_restore -U stellaops -d stellaops_authority \
--single-transaction --exit-on-error --no-owner --no-privileges \
"$AUTH9_DUMP_IN_CONTAINER"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres rm -f "$AUTH9_DUMP_IN_CONTAINER"
fi
auth9_issuer_fingerprints stellaops_platform "$AUTH9_WORK_DIR/issuer-source.after.tsv"
auth9_issuer_fingerprints stellaops_authority "$AUTH9_WORK_DIR/issuer-target.after.tsv"
auth9_issuer_audit_counts stellaops_platform "$AUTH9_WORK_DIR/issuer-source-audit.after.txt"
auth9_issuer_audit_counts stellaops_authority "$AUTH9_WORK_DIR/issuer-target-audit.after.txt"
diff -u "$AUTH9_WORK_DIR/issuer-source.after.tsv" "$AUTH9_WORK_DIR/issuer-target.after.tsv"
printf '%s\n' "copy_disposition=$AUTH9_COPY_DISPOSITION" \
"audit_source_before=$(cat "$AUTH9_WORK_DIR/issuer-source-audit.before.txt")" \
"audit_target_before=$(cat "$AUTH9_WORK_DIR/issuer-target-audit.before.txt")" \
"audit_source_after=$(cat "$AUTH9_WORK_DIR/issuer-source-audit.after.txt")" \
"audit_target_after=$(cat "$AUTH9_WORK_DIR/issuer-target-audit.after.txt")" \
> "$AUTH9_WORK_DIR/issuer-copy-record.txt"
7. Render last-wins, start the fold, and recreate only recorded callers
This helper reconstructs a stopped container from its own labels. It never borrows another service’s chain. The protected env file is explicit so a shell’s current directory cannot select a different .env. Before running the block, set these three non-secret window values in $AUTH9_ENV_FILE alongside the protected Authority connection changed in section 5:
AUTHORITY_ISSUER_DIRECTORY_FOLD_ENABLED=true
AUTHORITY_DOCTOR_REGISTRATION_ENABLED=true
AUTHORITY_ISSUER_DIRECTORY_DIRECT_BASE_ADDRESS=http://authority.stella-ops.local
Do not export them only in the interactive shell: Compose interpolation must be reproducible from the recorded protected env file.
auth9_compose_args_for() {
local container="$1" path
AUTH9_COMPOSE_PROJECT="$(docker inspect "$container" --format '{{index .Config.Labels "com.docker.compose.project"}}')"
AUTH9_COMPOSE_SERVICE="$(docker inspect "$container" --format '{{index .Config.Labels "com.docker.compose.service"}}')"
AUTH9_COMPOSE_WORKING_DIR="$(docker inspect "$container" --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}')"
AUTH9_COMPOSE_ARGS=(compose --project-name "$AUTH9_COMPOSE_PROJECT" \
--project-directory "$AUTH9_COMPOSE_WORKING_DIR" --env-file "$AUTH9_ENV_FILE")
while IFS= read -r path; do AUTH9_COMPOSE_ARGS+=(-f "$path"); done \
< "$AUTH9_WORK_DIR/${container}.config-files"
}
auth9_compose_args_for stellaops-authority
AUTH9_FOLD_OVERLAY="$(pwd)/devops/compose/docker-compose.authority.yml"
AUTH9_COMPOSE_ARGS+=(-f "$AUTH9_FOLD_OVERLAY" -f "$AUTH9_FINAL_PIN")
rendered_image="$(docker "${AUTH9_COMPOSE_ARGS[@]}" config --format json \
| jq -er '.services.authority.image')"
[[ "$rendered_image" == "$AUTH9_IMAGE_ID" ]] || {
echo "NO-GO: last-wins render selected $rendered_image, expected immutable $AUTH9_IMAGE_ID" >&2; exit 1;
}
docker "${AUTH9_COMPOSE_ARGS[@]}" config --format json | jq -e \
'.services.authority.environment.ISSUERDIRECTORY__FOLD__ENABLED == "true" and
.services.authority.environment["Doctor__Registration__Enabled"] == "true" and
.services.authority.environment["Doctor__Registration__PlatformBaseAddress"] == "http://platform.stella-ops.local" and
.services.authority.environment["Doctor__Registration__SelfEndpoint"] == "http://authority.stella-ops.local/doctor/authority/checks" and
.services.authority.environment["Doctor__Registration__Tenant"] == "default" and
.services.authority.pull_policy == "never"' \
>/dev/null
AUTH9_AUTHORITY_RECREATE_SINCE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
docker "${AUTH9_COMPOSE_ARGS[@]}" up -d --force-recreate --no-deps authority
[[ "$(docker inspect stellaops-authority --format '{{.Image}}')" == "$AUTH9_IMAGE_ID" ]]
docker exec stellaops-authority sh -lc 'test -s /app/buildinfo.json'
for _ in $(seq 1 60); do
[[ "$(docker inspect stellaops-authority --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}')" == healthy ]] && break
sleep 1
done
[[ "$(docker inspect stellaops-authority --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}')" == healthy ]]
for container in "${AUTH9_RUNNING_CALLERS[@]}"; do
auth9_compose_args_for "$container"
case "$container" in
stellaops-platform) env_key='STELLAOPS_ISSUERDIRECTORY_URL' ;;
*) env_key='IssuerDirectory__Client__BaseAddress' ;;
esac
rendered_target="$(docker "${AUTH9_COMPOSE_ARGS[@]}" config --format json \
| jq -er --arg service "$AUTH9_COMPOSE_SERVICE" --arg key "$env_key" \
'.services[$service].environment[$key]')"
[[ "$rendered_target" == 'http://authority.stella-ops.local' ]] || {
echo "NO-GO: $container renders $env_key=$rendered_target" >&2; exit 1;
}
docker "${AUTH9_COMPOSE_ARGS[@]}" up -d --force-recreate --no-deps "$AUTH9_COMPOSE_SERVICE"
docker inspect "$container" --format '{{range .Config.Env}}{{println .}}{{end}}' \
| grep -Fx "${env_key}=http://authority.stella-ops.local"
done
The loop contains only the runtime-discovered callers, so an absent Excititor role stays absent while Policy/ReleaseOrchestrator are included when their running environment actually opts in.
8. Route, issuer forcing pair, and atomic legacy stop
Require operator tokens only in memory. Never write them to the record. AUTH9_ISSUER_TOKEN must carry issuer-directory:read, issuer-directory:write, issuer-directory:admin, and the selected tenant; AUTH9_LOW_PRIVILEGE_TOKEN must be valid but omit issuer-directory:read.
read -r -s -p 'AUTH-9 issuer admin token: ' AUTH9_ISSUER_TOKEN; printf '\n'
read -r -s -p 'AUTH-9 low-privilege token: ' AUTH9_LOW_PRIVILEGE_TOKEN; printf '\n'
test -n "$AUTH9_ISSUER_TOKEN" && test -n "$AUTH9_LOW_PRIVILEGE_TOKEN"
: "${AUTH9_CANARY_TENANT:?set the token tenant slug}"
direct_code="$(docker exec stellaops-authority bash -lc \
"exec 3<>/dev/tcp/127.0.0.1/80; printf 'GET /issuer-directory/issuers HTTP/1.1\\r\\nHost: authority.stella-ops.local\\r\\nConnection: close\\r\\n\\r\\n' >&3; IFS=' ' read -r _ code _ <&3; printf '%s' \"\$code\"")"
[[ "$direct_code" == 401 || "$direct_code" == 403 ]]
set +e
route_precheck="$(pwsh tools/scripts/auth5-swap-issuer-directory-routes.ps1 -Check 2>&1)"
route_precheck_status=$?
set -e
printf '%s\n' "$route_precheck" | tee "$AUTH9_WORK_DIR/route-cutover-precheck.txt"
[[ "$route_precheck_status" == 1 ]]
[[ "$(grep -c ' PENDING /issuer-directory' <<< "$route_precheck")" == 2 ]]
grep -Fq -- '-Check: 2 config(s) would change. Nothing written.' <<< "$route_precheck"
pwsh tools/scripts/auth5-swap-issuer-directory-routes.ps1
pwsh tools/scripts/auth5-swap-issuer-directory-routes.ps1 -Check \
| tee "$AUTH9_WORK_DIR/route-cutover-applied.txt"
docker start stellaops-router-gateway >/dev/null
unauth_code="$(curl -sk -o /dev/null -w '%{http_code}' \
https://stella-ops.local/issuer-directory/issuers)"
low_code="$(curl -sk -o /dev/null -w '%{http_code}' \
-H "Authorization: Bearer $AUTH9_LOW_PRIVILEGE_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_CANARY_TENANT" \
https://stella-ops.local/issuer-directory/issuers)"
[[ "$unauth_code" == 401 || "$unauth_code" == 403 ]]
[[ "$low_code" == 403 ]]
# THE CANARY ID MUST BE A UUID, NOT A WINDOW SLUG. `issuer.issuers.id` is
# `UUID PRIMARY KEY` (migration 024 line 65) and PostgresIssuerRepository
# Guid.Parses the issuer id on both write paths (Write.cs:25 upsert,
# Write.cs:109 delete), so a slug-shaped id answers 500
# `System.FormatException: Unrecognized Guid format` from the endpoint, never
# 201 -- on the standalone Postgres host as well as the folded one. The slug
# form was never reached before: window 2 stopped at 400 tenant_missing ahead
# of persistence (AUTH-21). Measured on the window-3 image in its pre-flight
# container smoke, 2026-09-08: slug -> 500 on create AND on delete; UUID -> 201
# create, 200 read, 204 delete, 404 after delete. The readable window label
# stays where it belongs, in displayName and slug, so the canary is still
# identifiable in the audit rows and in issuer.issuers.name.
AUTH9_ISSUER_CANARY="$(od -An -tx1 -N16 /dev/urandom | tr -d ' \n' \
| sed -E 's/^(.{8})(.{4})(.{3})(.)(.{3})(.*)$/\1-\2-4\3-a\5-\6/' | cut -c1-36)"
[[ "$AUTH9_ISSUER_CANARY" =~ ^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-a[0-9a-f]{3}-[0-9a-f]{12}$ ]]
AUTH9_ISSUER_CANARY_LABEL="auth9-${AUTH9_WINDOW_ID}-$(date -u +%Y%m%dT%H%M%SZ)"
jq -n --arg id "$AUTH9_ISSUER_CANARY" --arg label "$AUTH9_ISSUER_CANARY_LABEL" \
'{id:$id,displayName:$label,slug:$label}' > "$AUTH9_WORK_DIR/issuer-canary-request.json"
create_code="$(curl -sk -o "$AUTH9_WORK_DIR/issuer-canary-created.json" -w '%{http_code}' \
-X POST -H "Authorization: Bearer $AUTH9_ISSUER_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_CANARY_TENANT" \
-H "X-StellaOps-Reason: AUTH-9 ${AUTH9_WINDOW_ID} forcing probe" \
-H 'Content-Type: application/json' \
--data-binary @"$AUTH9_WORK_DIR/issuer-canary-request.json" \
https://stella-ops.local/issuer-directory/issuers)"
[[ "$create_code" == 201 ]]
read_code="$(curl -sk -o "$AUTH9_WORK_DIR/issuer-canary-read.json" -w '%{http_code}' \
-H "Authorization: Bearer $AUTH9_ISSUER_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_CANARY_TENANT" \
"https://stella-ops.local/issuer-directory/issuers/$AUTH9_ISSUER_CANARY")"
[[ "$read_code" == 200 ]]
jq -e --arg id "$AUTH9_ISSUER_CANARY" '.id == $id' \
< "$AUTH9_WORK_DIR/issuer-canary-read.json" >/dev/null
# Exercise the trust evaluator, not only issuer CRUD. The write is tenant-bound
# by the token claim; the header is a consistency witness, never the authority.
jq -n --arg reason "AUTH-9 ${AUTH9_WINDOW_ID} trust forcing probe" \
'{weight:0.75,reason:$reason}' > "$AUTH9_WORK_DIR/issuer-trust-request.json"
trust_set_code="$(curl -sk -o "$AUTH9_WORK_DIR/issuer-trust-set.json" -w '%{http_code}' \
-X PUT -H "Authorization: Bearer $AUTH9_ISSUER_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_CANARY_TENANT" \
-H "X-StellaOps-Reason: AUTH-9 ${AUTH9_WINDOW_ID} trust forcing probe" \
-H 'Content-Type: application/json' \
--data-binary @"$AUTH9_WORK_DIR/issuer-trust-request.json" \
"https://stella-ops.local/issuer-directory/issuers/$AUTH9_ISSUER_CANARY/trust")"
[[ "$trust_set_code" == 200 ]]
jq -e '.tenantOverride.weight == 0.75 and .effectiveWeight == 0.75' \
< "$AUTH9_WORK_DIR/issuer-trust-set.json" >/dev/null
trust_read_code="$(curl -sk -o "$AUTH9_WORK_DIR/issuer-trust-read.json" -w '%{http_code}' \
-H "Authorization: Bearer $AUTH9_ISSUER_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_CANARY_TENANT" \
"https://stella-ops.local/issuer-directory/issuers/$AUTH9_ISSUER_CANARY/trust?includeGlobal=true")"
[[ "$trust_read_code" == 200 ]]
jq -e '.tenantOverride.weight == 0.75 and .effectiveWeight == 0.75' \
< "$AUTH9_WORK_DIR/issuer-trust-read.json" >/dev/null
trust_delete_code="$(curl -sk -o /dev/null -w '%{http_code}' -X DELETE \
-H "Authorization: Bearer $AUTH9_ISSUER_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_CANARY_TENANT" \
-H "X-StellaOps-Reason: AUTH-9 ${AUTH9_WINDOW_ID} trust forcing cleanup" \
"https://stella-ops.local/issuer-directory/issuers/$AUTH9_ISSUER_CANARY/trust")"
[[ "$trust_delete_code" == 204 ]]
trust_gone_code="$(curl -sk -o "$AUTH9_WORK_DIR/issuer-trust-gone.json" -w '%{http_code}' \
-H "Authorization: Bearer $AUTH9_ISSUER_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_CANARY_TENANT" \
"https://stella-ops.local/issuer-directory/issuers/$AUTH9_ISSUER_CANARY/trust?includeGlobal=true")"
[[ "$trust_gone_code" == 200 ]]
jq -e '.tenantOverride == null' "$AUTH9_WORK_DIR/issuer-trust-gone.json" >/dev/null
delete_code="$(curl -sk -o /dev/null -w '%{http_code}' -X DELETE \
-H "Authorization: Bearer $AUTH9_ISSUER_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_CANARY_TENANT" \
-H "X-StellaOps-Reason: AUTH-9 ${AUTH9_WINDOW_ID} forcing cleanup" \
"https://stella-ops.local/issuer-directory/issuers/$AUTH9_ISSUER_CANARY")"
[[ "$delete_code" == 204 ]]
gone_code="$(curl -sk -o /dev/null -w '%{http_code}' \
-H "Authorization: Bearer $AUTH9_ISSUER_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_CANARY_TENANT" \
"https://stella-ops.local/issuer-directory/issuers/$AUTH9_ISSUER_CANARY")"
[[ "$gone_code" == 404 ]]
docker stop stellaops-issuer-directory >/dev/null
test -z "$(docker ps --filter 'name=^/stellaops-issuer-directory$' --format '{{.Names}}')"
The two token variables must be unset after the soak or immediately on rollback.
9. Tenant gate, permission-error soak, and fence release
Owner ruling 2026-08-23: use one reversible update; do not add a tenant-delete endpoint. AUTH9-TENANT-DISPLAY-NAME-REVERSIBLE-V1 updates only the display name of an owner-designated existing disposable tenant, proves the source row, same-transaction outbox event, canonical Timeline audit, and Policy’s already live local replica/checkpoint, then restores the exact prior display name through the same API and proves the restoration everywhere. authority.tenants.row_version is the concurrency witness: the update must be exactly before + 1 and the restore exactly before + 2. The procedure refuses default, a suspended tenant, a stale Policy replica, or any unexpected version jump. The endpoint changes the admin-facing name only and preserves the independent display_name catalog label byte-for-byte, so a tenant whose two values already differ is now a safe reversible forcing target.
The admin token must be freshly minted (the endpoint’s auth_time window is five minutes), scoped to $AUTH9_DISPOSABLE_TENANT, and carry ui.admin, authority:tenants.read, and authority:tenants.write. Keep it only in memory. Policy is the named consumer because its catalog.replica.tenants drain is already live in stellaops_policy; substituting an unapproved replica or merely checking row count is a NO-GO.
Before opening a live window, run TenantPropagationEndToEndTests.ReversibleDisplayNameUpdate_RestoresExactValue_WithVersionedAuditAndReplicaProof. It drives the real PATCH endpoint twice against scratch PostgreSQL, proves exact +1/+2 source and outbox versions, captures the canonical correlated update_tenant audit payload, drains the standard catalog consumer into a separate scratch database named stellaops_policy, verifies its catalog.replica.tenants checkpoint, and finishes on the exact prior value. This is source/image capability evidence only: it does not replace the live Policy container, Timeline persistence, owner-designated tenant, or window forcing proof below.
auth9_require_actual AUTH9_TENANT_FORCING_PROCEDURE_ID
[[ "$AUTH9_TENANT_FORCING_PROCEDURE_ID" == 'AUTH9-TENANT-DISPLAY-NAME-REVERSIBLE-V1' ]]
auth9_require_actual AUTH9_DISPOSABLE_TENANT
auth9_require_actual AUTH9_DISPOSABLE_TENANT_APPROVAL
[[ "$AUTH9_DISPOSABLE_TENANT" =~ ^[a-z0-9][a-z0-9-]*$ ]]
[[ "$AUTH9_DISPOSABLE_TENANT" != default ]]
AUTH9_TENANT_REPLICA_CONTAINER=stellaops-policy-engine
AUTH9_TENANT_REPLICA_DATABASE=stellaops_policy
AUTH9_TENANT_REPLICA_CONSUMER=catalog.replica.tenants
AUTH9_TENANT_REPLICA_STREAM=tenants
# SPRINT_20260722_016 AUTH-25 (2026-09-08): `shared.tenants` in
# `stellaops_platform` is written by PLATFORM now, not by Authority. Authority
# holds no connection to that database, so the legacy half of this forcing pair
# converges through a DRAIN rather than through a second write inside the PATCH.
# platform-web is therefore a PARTICIPANT in this window, and its replication
# must be on before the pair runs or the shared row never moves at all.
AUTH9_SHARED_TENANTS_OWNER_CONTAINER=stellaops-platform
[[ "$(docker inspect "$AUTH9_TENANT_REPLICA_CONTAINER" \
--format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}')" == healthy ]]
docker inspect "$AUTH9_TENANT_REPLICA_CONTAINER" \
--format '{{range .Config.Env}}{{println .}}{{end}}' \
| grep -Fx 'Catalog__Replication__Tenants__Enabled=true'
[[ "$(docker inspect "$AUTH9_SHARED_TENANTS_OWNER_CONTAINER" \
--format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}')" == healthy ]]
docker inspect "$AUTH9_SHARED_TENANTS_OWNER_CONTAINER" \
--format '{{range .Config.Env}}{{println .}}{{end}}' \
| grep -Fx 'Catalog__Replication__Tenants__Enabled=true'
auth9_source_tenant_json() {
docker exec stellaops-postgres psql -XqAt -U stellaops \
-d stellaops_authority -c \
"SELECT jsonb_build_object(
'id', id::text, 'tenantId', tenant_id, 'name', name,
'displayName', display_name, 'status', status,
'rowVersion', row_version, 'updatedAt', updated_at)::text
FROM authority.tenants WHERE tenant_id = '$AUTH9_DISPOSABLE_TENANT';"
}
auth9_shared_tenant_json() {
docker exec stellaops-postgres psql -XqAt -U stellaops \
-d stellaops_platform -c \
"SELECT jsonb_build_object(
'id', id::text, 'tenantId', tenant_id, 'name', name,
'displayName', display_name, 'status', status)::text
FROM shared.tenants WHERE tenant_id = '$AUTH9_DISPOSABLE_TENANT';"
}
auth9_require_no_api_warning() {
local headers="$1"
if grep -Eqi '^Warning:' "$headers"; then
echo "NO-GO: tenant PATCH returned an API Warning header. Since AUTH-25 the mutation has ONE half -- the canonical row and its catalog event, in one transaction -- so a Warning here is NOT a propagation partial success. It is either a client-grant failure or a re-introduced cross-database writer. Read the header text and the audit events before continuing" >&2
grep -Ei '^Warning:' "$headers" >&2
return 1
fi
}
auth9_policy_tenant_json() {
docker exec stellaops-postgres psql -XqAt -U stellaops \
-d "$AUTH9_TENANT_REPLICA_DATABASE" \
-c \
"SELECT jsonb_build_object(
'replica', (SELECT jsonb_build_object(
'catalog', catalog, 'key', key, 'version', version,
'payload', payload, 'deleted', deleted)
FROM catalog_replica.replica_rows
WHERE catalog = 'tenants' AND key = '$AUTH9_DISPOSABLE_TENANT'),
'checkpoint', (SELECT jsonb_build_object(
'consumerId', consumer_id, 'stream', stream,
'streamEpoch', stream_epoch, 'seq', seq, 'updatedAt', updated_at)
FROM eventing.consumer_checkpoints
WHERE consumer_id = '$AUTH9_TENANT_REPLICA_CONSUMER' AND stream = '$AUTH9_TENANT_REPLICA_STREAM'))::text;"
}
auth9_tenant_outbox_json() {
local version="$1"
docker exec stellaops-postgres psql -XqAt -U stellaops \
-d stellaops_authority \
-c \
"SELECT jsonb_build_object(
'streamEpoch', stream_epoch, 'seq', seq, 'eventId', event_id,
'version', (payload->>'row_version')::bigint,
'payload', payload->'payload',
'deleted', (payload->>'deleted')::boolean)::text
FROM eventing.outbox
WHERE stream = 'tenants' AND payload->>'key' = '$AUTH9_DISPOSABLE_TENANT'
AND (payload->>'row_version')::bigint = $version
ORDER BY seq DESC LIMIT 1;"
}
auth9_wait_policy_version() {
local version="$1" name="$2" display_name="$3" epoch="$4" seq="$5" output="$6"
for _ in $(seq 1 60); do
auth9_policy_tenant_json > "$output"
if jq -e --arg tenant "$AUTH9_DISPOSABLE_TENANT" --arg name "$name" \
--arg displayName "$display_name" \
--argjson version "$version" --argjson epoch "$epoch" --argjson seq "$seq" \
'.replica.catalog == "tenants" and .replica.key == $tenant and
.replica.version == $version and .replica.deleted == false and
.replica.payload.tenant_id == $tenant and
.replica.payload.name == $name and
.replica.payload.display_name == $displayName and
.checkpoint.consumerId == "catalog.replica.tenants" and
.checkpoint.stream == "tenants" and
.checkpoint.streamEpoch == $epoch and .checkpoint.seq >= $seq' \
< "$output" >/dev/null; then
return 0
fi
sleep 2
done
echo "NO-GO: Policy tenant replica/checkpoint did not converge to version $version" >&2
return 1
}
# SPRINT_20260722_016 AUTH-25 (2026-09-08). READ THE BOUND BEFORE CHANGING THE
# TIMEOUT. `shared.tenants` used to move INSIDE the PATCH, because Authority
# wrote it directly across the database boundary. It does not any more: Platform
# drains Authority's `tenants` catalog and projects it, so the shared row is
# EVENTUALLY consistent with a bound of one poll interval
# (`Catalog:Replication:Tenants:PollInterval`, default 15 s) plus the projection
# write. The separate 30 s `BootstrapTimeout` covers only a COLD platform-web
# start and does not apply mid-window. Reading the row immediately after the
# PATCH is therefore a race that passes on a fast machine and fails on a slow
# one -- the worst kind of window step. The 60x2 s ceiling below is the same
# shape auth9_wait_policy_version uses and leaves ~8x headroom over the bound.
auth9_wait_shared_tenant() {
local name="$1" display_name="$2" status="$3" output="$4"
for _ in $(seq 1 60); do
auth9_shared_tenant_json > "$output"
if jq -e --arg tenant "$AUTH9_DISPOSABLE_TENANT" --arg name "$name" \
--arg displayName "$display_name" --arg status "$status" \
'.tenantId == $tenant and .name == $name and
.displayName == $displayName and .status == $status' \
< "$output" >/dev/null; then
return 0
fi
sleep 2
done
echo "NO-GO: shared.tenants did not converge to name '$name' within 120 s." >&2
echo " This is a PLATFORM-side drain failure, never an Authority one: Authority" >&2
echo " holds no connection to stellaops_platform. Check platform-web's" >&2
echo " Catalog:Replication:Tenants drain -- feed reachable, catalog:replicate" >&2
echo " granted to its client, host started at all -- before suspecting the" >&2
echo " PATCH, whose own success the source row and the outbox already prove." >&2
return 1
}
auth9_guid_for() {
local hex
hex="$(printf '%s' "$AUTH9_WINDOW_ID/$1" | sha256sum | cut -c1-32)"
printf '%s-%s-%s-%s-%s' \
"${hex:0:8}" "${hex:8:4}" "${hex:12:4}" "${hex:16:4}" "${hex:20:12}"
}
# ACTION NAME CORRECTED, 2026-09-08 (window 4, the first live execution of this
# gate). This helper asked for `action = 'update_tenant'`, which matches nothing
# in `timeline.unified_audit_events` -- ZERO rows with that action, ever.
#
# ROOT CAUSE RE-DIAGNOSED, 2026-09-09 (AUTH-28). The first diagnosis said
# Authority never emits `update_tenant`. It does, and has since 2026-04-09. The
# tenant PATCH is registered
# `.Audited(AuditModules.Authority, AuditActions.Authority.UpdateTenant)` in
# `ConsoleAdminEndpointExtensions.cs`, and `AuditActionFilter` stamps that action
# verbatim. What the earlier reading found at that file's `:558` and `:648` is a
# DIFFERENT stream: an `action` PROPERTY inside the `IAuthEventSink` payload,
# which reaches Timeline as `resource_type = 'authority_session'`.
#
# The name is destroyed at INGEST, not at emission. Timeline's
# `UnifiedAuditValueMapper.NormalizeAction` returns the first
# `UnifiedAuditCatalog.Actions` entry that is a SUBSTRING of the raw action, and
# "update_tenant" contains "update". So `update` IS the right value to gate on
# here, for a sounder reason than the original one, and the specific name stays
# recoverable in `tags` (`tags @> '{update_tenant}'`).
#
# Do NOT "fix" `TenantPropagationEndToEndTests` to match this line. That test
# observes the EMITTER, where the action is and must remain `update_tenant`;
# changing it would make it permanently red. Both boundaries are pinned by
# `AuthorityConsolidationConformanceTests.Auth9TenantAuditActionName_IsPinnedAtBothBoundaries`,
# which reads this sheet with its comment lines stripped -- the predecessor guard
# matched the whole file and was satisfied by this very comment quoting the old
# name, so it kept passing while pinning nothing.
#
# Windows 1-3 never reached this assertion -- window 3 stopped on the tenant
# Warning two steps earlier -- so a gate that could only ever fail sat here
# unexecuted. Window 4 hit it AFTER a committed mutation, which is the expensive
# place to find it: the temporary business value was live on a real tenant while
# the gate refused. The audit it was looking for was present the whole time,
# correlated and at statusCode 200.
#
# `resource_type = 'console'` is asserted beside the action so the narrowing the
# old name provided is not simply dropped: `update` alone also matches
# `authority_session` rows, of which this estate holds about 323k.
auth9_timeline_audit_json() {
local correlation="$1"
docker exec stellaops-postgres psql -XqAt -U stellaops \
-d stellaops_timeline \
-c \
"SELECT jsonb_build_object(
'id', id, 'tenantId', tenant_id, 'module', module, 'action', action,
'resourceType', resource_type, 'resourceId', resource_id,
'correlationId', correlation_id, 'statusCode', details_jsonb->>'statusCode')::text
FROM timeline.unified_audit_events
WHERE tenant_id = '$AUTH9_DISPOSABLE_TENANT' AND module = 'authority' AND action = 'update'
AND resource_type = 'console'
AND resource_id = '$AUTH9_DISPOSABLE_TENANT' AND correlation_id = '$correlation'
AND details_jsonb->>'statusCode' = '200'
ORDER BY timestamp DESC LIMIT 1;"
}
auth9_wait_timeline_audit() {
local correlation="$1" output="$2"
for _ in $(seq 1 30); do
auth9_timeline_audit_json "$correlation" > "$output"
if jq -e --arg tenant "$AUTH9_DISPOSABLE_TENANT" --arg correlation "$correlation" \
'.tenantId == $tenant and .module == "authority" and
.action == "update" and .resourceType == "console" and
.resourceId == $tenant and
.correlationId == $correlation and .statusCode == "200"' \
< "$output" >/dev/null; then
return 0
fi
sleep 2
done
echo "NO-GO: canonical Timeline audit did not record correlation $correlation" >&2
return 1
}
AUTH9_TENANT_BEFORE_SOURCE="$AUTH9_WORK_DIR/tenant-before-source.json"
AUTH9_TENANT_BEFORE_SHARED="$AUTH9_WORK_DIR/tenant-before-shared.json"
AUTH9_TENANT_BEFORE_REPLICA="$AUTH9_WORK_DIR/tenant-before-policy-replica.json"
auth9_source_tenant_json > "$AUTH9_TENANT_BEFORE_SOURCE"
auth9_shared_tenant_json > "$AUTH9_TENANT_BEFORE_SHARED"
auth9_policy_tenant_json > "$AUTH9_TENANT_BEFORE_REPLICA"
jq -e --arg tenant "$AUTH9_DISPOSABLE_TENANT" \
'.tenantId == $tenant and .status == "active" and
(.name | type == "string" and length > 0) and
(.displayName | type == "string" and length > 0) and
(.rowVersion | type == "number" and . > 0)' \
< "$AUTH9_TENANT_BEFORE_SOURCE" >/dev/null
test -s "$AUTH9_TENANT_BEFORE_SHARED"
jq -S '{id,tenantId,name,displayName,status}' "$AUTH9_TENANT_BEFORE_SOURCE" \
> "$AUTH9_WORK_DIR/tenant-before-source-shared-projection.json"
jq -S . "$AUTH9_TENANT_BEFORE_SHARED" \
> "$AUTH9_WORK_DIR/tenant-before-shared-sorted.json"
diff -u "$AUTH9_WORK_DIR/tenant-before-source-shared-projection.json" \
"$AUTH9_WORK_DIR/tenant-before-shared-sorted.json"
AUTH9_TENANT_BEFORE_NAME="$(jq -er '.name' < "$AUTH9_TENANT_BEFORE_SOURCE")"
AUTH9_TENANT_BEFORE_DISPLAY_NAME="$(jq -er '.displayName' < "$AUTH9_TENANT_BEFORE_SOURCE")"
AUTH9_TENANT_BEFORE_VERSION="$(jq -er '.rowVersion' < "$AUTH9_TENANT_BEFORE_SOURCE")"
jq -e --arg tenant "$AUTH9_DISPOSABLE_TENANT" --arg name "$AUTH9_TENANT_BEFORE_NAME" \
--arg displayName "$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
--argjson version "$AUTH9_TENANT_BEFORE_VERSION" \
'.replica.key == $tenant and .replica.version == $version and
.replica.deleted == false and .replica.payload.tenant_id == $tenant and
.replica.payload.name == $name and
.replica.payload.display_name == $displayName and
.checkpoint.consumerId == "catalog.replica.tenants" and
.checkpoint.stream == "tenants"' \
< "$AUTH9_TENANT_BEFORE_REPLICA" >/dev/null
read -r -s -p 'AUTH-9 fresh tenant-admin token: ' AUTH9_TENANT_ADMIN_TOKEN; printf '\n'
test -n "$AUTH9_TENANT_ADMIN_TOKEN"
before_api_code="$(curl -sk -o "$AUTH9_WORK_DIR/tenant-before-api.json" -w '%{http_code}' \
-H "Authorization: Bearer $AUTH9_TENANT_ADMIN_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_DISPOSABLE_TENANT" \
https://stella-ops.local/console/admin/tenants)"
[[ "$before_api_code" == 200 ]]
jq -e --arg tenant "$AUTH9_DISPOSABLE_TENANT" --arg name "$AUTH9_TENANT_BEFORE_NAME" \
'[.tenants[] | select(.id == $tenant and .displayName == $name)] | length == 1' \
< "$AUTH9_WORK_DIR/tenant-before-api.json" >/dev/null
AUTH9_TENANT_TEMP_NAME="AUTH-9 reversible ${AUTH9_WINDOW_ID}"
[[ "$AUTH9_TENANT_TEMP_NAME" != "$AUTH9_TENANT_BEFORE_NAME" ]]
AUTH9_TENANT_UPDATE_VERSION="$((AUTH9_TENANT_BEFORE_VERSION + 1))"
AUTH9_TENANT_RESTORE_VERSION="$((AUTH9_TENANT_BEFORE_VERSION + 2))"
AUTH9_TENANT_UPDATE_CORRELATION_ID="$(auth9_guid_for tenant-update)"
AUTH9_TENANT_RESTORE_CORRELATION_ID="$(auth9_guid_for tenant-restore)"
jq -n --arg displayName "$AUTH9_TENANT_TEMP_NAME" \
'{displayName:$displayName}' > "$AUTH9_WORK_DIR/tenant-update-request.json"
jq -n --arg displayName "$AUTH9_TENANT_BEFORE_NAME" \
'{displayName:$displayName}' > "$AUTH9_WORK_DIR/tenant-restore-request.json"
AUTH9_TENANT_RESTORE_REQUIRED=0
# AUTH-25: the restore reaches the shared row through the SAME drain, so this
# waits too. It still finishes on an exact diff against the captured pre-state:
# the wait decides WHEN to compare, never WHAT counts as equal.
auth9_assert_shared_restored() {
local output="$1"
auth9_wait_shared_tenant "$AUTH9_TENANT_BEFORE_NAME" \
"$AUTH9_TENANT_BEFORE_DISPLAY_NAME" active "$output"
jq -S . "$output" > "${output%.json}.sorted.json"
diff -u "$AUTH9_WORK_DIR/tenant-before-shared-sorted.json" \
"${output%.json}.sorted.json"
}
auth9_restore_tenant_value() {
local restore_code current_source="$AUTH9_WORK_DIR/tenant-before-restore-source.json"
auth9_source_tenant_json > "$current_source"
if jq -e --arg name "$AUTH9_TENANT_BEFORE_NAME" \
--arg displayName "$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
--argjson version "$AUTH9_TENANT_BEFORE_VERSION" \
'.name == $name and .displayName == $displayName and .rowVersion == $version' \
< "$current_source" >/dev/null; then
# The update request did not commit; there is nothing to undo.
cp "$current_source" "$AUTH9_WORK_DIR/tenant-restored-source.json"
auth9_assert_shared_restored "$AUTH9_WORK_DIR/tenant-restored-shared.json"
AUTH9_TENANT_RESTORE_REQUIRED=0
return 0
fi
if jq -e --arg name "$AUTH9_TENANT_BEFORE_NAME" \
--arg displayName "$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
--argjson version "$AUTH9_TENANT_RESTORE_VERSION" \
'.name == $name and .displayName == $displayName and .rowVersion == $version' \
< "$current_source" >/dev/null; then
# A prior restore request committed even if its response was lost.
cp "$current_source" "$AUTH9_WORK_DIR/tenant-restored-source.json"
auth9_assert_shared_restored "$AUTH9_WORK_DIR/tenant-restored-shared.json"
AUTH9_TENANT_RESTORE_REQUIRED=0
return 0
fi
if ! jq -e --arg name "$AUTH9_TENANT_TEMP_NAME" \
--arg displayName "$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
--argjson version "$AUTH9_TENANT_UPDATE_VERSION" \
'.name == $name and .displayName == $displayName and .rowVersion == $version' \
< "$current_source" >/dev/null; then
echo 'CRITICAL: tenant state/version changed unexpectedly; refusing to overwrite it; keep the maintenance fence and escalate' >&2
return 1
fi
restore_code="$(curl -sk -D "$AUTH9_WORK_DIR/tenant-restored-headers.txt" \
-o "$AUTH9_WORK_DIR/tenant-restored-api.json" -w '%{http_code}' \
-X PATCH -H "Authorization: Bearer $AUTH9_TENANT_ADMIN_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_DISPOSABLE_TENANT" \
-H "X-StellaOps-Correlation-Id: $AUTH9_TENANT_RESTORE_CORRELATION_ID" \
-H "X-StellaOps-Reason: AUTH-9 ${AUTH9_WINDOW_ID} restore exact prior tenant display name" \
-H 'Content-Type: application/json' \
--data-binary @"$AUTH9_WORK_DIR/tenant-restore-request.json" \
"https://stella-ops.local/console/admin/tenants/$AUTH9_DISPOSABLE_TENANT" || true)"
[[ "$restore_code" == 200 ]]
auth9_require_no_api_warning "$AUTH9_WORK_DIR/tenant-restored-headers.txt"
jq -e --arg tenant "$AUTH9_DISPOSABLE_TENANT" --arg name "$AUTH9_TENANT_BEFORE_NAME" \
'.id == $tenant and .displayName == $name' \
< "$AUTH9_WORK_DIR/tenant-restored-api.json" >/dev/null
auth9_source_tenant_json > "$AUTH9_WORK_DIR/tenant-restored-source.json"
jq -e --arg name "$AUTH9_TENANT_BEFORE_NAME" \
--arg displayName "$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
--argjson version "$AUTH9_TENANT_RESTORE_VERSION" \
'.name == $name and .displayName == $displayName and
.status == "active" and .rowVersion == $version' \
< "$AUTH9_WORK_DIR/tenant-restored-source.json" >/dev/null
auth9_assert_shared_restored "$AUTH9_WORK_DIR/tenant-restored-shared.json"
AUTH9_TENANT_RESTORE_REQUIRED=0
}
auth9_tenant_exit() {
local original_status="$?" restore_status=0
trap - EXIT INT TERM
if [[ "${AUTH9_TENANT_RESTORE_REQUIRED:-0}" == 1 ]]; then
echo 'AUTH-9: failure after tenant mutation; restoring the captured display name before exit' >&2
set +e
auth9_restore_tenant_value
restore_status="$?"
set -e
if [[ "$restore_status" != 0 ]]; then
echo 'CRITICAL: automatic tenant restoration failed; keep the maintenance fence and escalate' >&2
fi
fi
exit "$original_status"
}
trap auth9_tenant_exit EXIT
trap 'exit 130' INT TERM
AUTH9_TENANT_RESTORE_REQUIRED=1
update_code="$(curl -sk -D "$AUTH9_WORK_DIR/tenant-updated-headers.txt" \
-o "$AUTH9_WORK_DIR/tenant-updated-api.json" -w '%{http_code}' \
-X PATCH -H "Authorization: Bearer $AUTH9_TENANT_ADMIN_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_DISPOSABLE_TENANT" \
-H "X-StellaOps-Correlation-Id: $AUTH9_TENANT_UPDATE_CORRELATION_ID" \
-H "X-StellaOps-Reason: AUTH-9 ${AUTH9_WINDOW_ID} reversible tenant forcing update" \
-H 'Content-Type: application/json' \
--data-binary @"$AUTH9_WORK_DIR/tenant-update-request.json" \
"https://stella-ops.local/console/admin/tenants/$AUTH9_DISPOSABLE_TENANT" || true)"
auth9_source_tenant_json > "$AUTH9_WORK_DIR/tenant-after-update-source.json"
if jq -e --arg name "$AUTH9_TENANT_TEMP_NAME" \
--arg displayName "$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
--argjson version "$AUTH9_TENANT_UPDATE_VERSION" \
'.name == $name and .displayName == $displayName and .rowVersion == $version' \
< "$AUTH9_WORK_DIR/tenant-after-update-source.json" >/dev/null; then
: # The failure trap now has an exact, version-checked state it may restore.
elif jq -e --arg name "$AUTH9_TENANT_BEFORE_NAME" \
--arg displayName "$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
--argjson version "$AUTH9_TENANT_BEFORE_VERSION" \
'.name == $name and .displayName == $displayName and .rowVersion == $version' \
< "$AUTH9_WORK_DIR/tenant-after-update-source.json" >/dev/null; then
AUTH9_TENANT_RESTORE_REQUIRED=0
else
echo 'CRITICAL: tenant state/version changed unexpectedly; refusing to overwrite it; keep the maintenance fence and escalate' >&2
exit 1
fi
# ORDER MATTERS HERE, and it was wrong until window 3 (2026-09-08). The endpoint
# can answer 200 WITH a `Warning:` header for an update that COMMITTED. Both
# assertions below must therefore run with the failure trap already armed
# (AUTH9_TENANT_RESTORE_REQUIRED=1 above) and after the state has been
# version-checked, so a warning unwinds through auth9_restore_tenant_value
# instead of exiting with the temporary business value live on a real tenant.
# Window 3 hit this for real: a `Warning:` fired on a committed update and the
# disposable tenant had to be restored by hand.
#
# KEEP THIS ORDER EVEN THOUGH ITS ORIGINAL CAUSE IS GONE. What window 3 hit was
# `199 ... tenant propagation partial success`, and AUTH-25 deleted the writer
# that emitted it -- a tenant mutation now has one half, so THAT warning cannot
# recur. The ordering still earns its place for two reasons: the same endpoint
# emits OTHER 199 warnings on a committed update (the client-grant failures in
# docs/runbooks/general/tenant-onboarding.md), and a warning nobody predicted is
# exactly the case where unwinding cleanly matters most. Treat a `Warning:` here
# as a NO-GO to investigate, not as a known propagation state.
[[ "$update_code" == 200 ]]
auth9_require_no_api_warning "$AUTH9_WORK_DIR/tenant-updated-headers.txt"
jq -e --arg tenant "$AUTH9_DISPOSABLE_TENANT" --arg name "$AUTH9_TENANT_TEMP_NAME" \
'.id == $tenant and .displayName == $name' \
< "$AUTH9_WORK_DIR/tenant-updated-api.json" >/dev/null
# AUTH-25: WAIT for the drain rather than reading the row now. The PATCH is
# already proven by the source row and the outbox above; what this step
# measures is that the OWNER converged its projection from that event.
auth9_wait_shared_tenant "$AUTH9_TENANT_TEMP_NAME" \
"$AUTH9_TENANT_BEFORE_DISPLAY_NAME" active \
"$AUTH9_WORK_DIR/tenant-after-update-shared.json"
auth9_tenant_outbox_json "$AUTH9_TENANT_UPDATE_VERSION" \
> "$AUTH9_WORK_DIR/tenant-update-outbox.json"
AUTH9_TENANT_UPDATE_EPOCH="$(jq -er '.streamEpoch' < "$AUTH9_WORK_DIR/tenant-update-outbox.json")"
AUTH9_TENANT_UPDATE_SEQ="$(jq -er '.seq' < "$AUTH9_WORK_DIR/tenant-update-outbox.json")"
jq -e --arg name "$AUTH9_TENANT_TEMP_NAME" \
--arg displayName "$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
--argjson version "$AUTH9_TENANT_UPDATE_VERSION" \
'.version == $version and .deleted == false and
.payload.name == $name and .payload.display_name == $displayName' \
< "$AUTH9_WORK_DIR/tenant-update-outbox.json" >/dev/null
auth9_wait_policy_version "$AUTH9_TENANT_UPDATE_VERSION" "$AUTH9_TENANT_TEMP_NAME" \
"$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
"$AUTH9_TENANT_UPDATE_EPOCH" "$AUTH9_TENANT_UPDATE_SEQ" \
"$AUTH9_WORK_DIR/tenant-updated-policy-replica.json"
auth9_wait_timeline_audit "$AUTH9_TENANT_UPDATE_CORRELATION_ID" \
"$AUTH9_WORK_DIR/tenant-update-timeline-audit.json"
# Restore immediately; the soak never runs with the temporary business value.
auth9_restore_tenant_value
jq -e --argjson version "$AUTH9_TENANT_RESTORE_VERSION" \
'.rowVersion == $version' < "$AUTH9_WORK_DIR/tenant-restored-source.json" >/dev/null
auth9_tenant_outbox_json "$AUTH9_TENANT_RESTORE_VERSION" \
> "$AUTH9_WORK_DIR/tenant-restore-outbox.json"
AUTH9_TENANT_RESTORE_EPOCH="$(jq -er '.streamEpoch' < "$AUTH9_WORK_DIR/tenant-restore-outbox.json")"
AUTH9_TENANT_RESTORE_SEQ="$(jq -er '.seq' < "$AUTH9_WORK_DIR/tenant-restore-outbox.json")"
jq -e --arg name "$AUTH9_TENANT_BEFORE_NAME" \
--arg displayName "$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
--argjson version "$AUTH9_TENANT_RESTORE_VERSION" \
'.version == $version and .deleted == false and
.payload.name == $name and .payload.display_name == $displayName' \
< "$AUTH9_WORK_DIR/tenant-restore-outbox.json" >/dev/null
auth9_wait_policy_version "$AUTH9_TENANT_RESTORE_VERSION" "$AUTH9_TENANT_BEFORE_NAME" \
"$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
"$AUTH9_TENANT_RESTORE_EPOCH" "$AUTH9_TENANT_RESTORE_SEQ" \
"$AUTH9_WORK_DIR/tenant-restored-policy-replica.json"
auth9_wait_timeline_audit "$AUTH9_TENANT_RESTORE_CORRELATION_ID" \
"$AUTH9_WORK_DIR/tenant-restore-timeline-audit.json"
restored_api_code="$(curl -sk -o "$AUTH9_WORK_DIR/tenant-restored-list-api.json" -w '%{http_code}' \
-H "Authorization: Bearer $AUTH9_TENANT_ADMIN_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_DISPOSABLE_TENANT" \
https://stella-ops.local/console/admin/tenants)"
[[ "$restored_api_code" == 200 ]]
jq -e --arg tenant "$AUTH9_DISPOSABLE_TENANT" --arg name "$AUTH9_TENANT_BEFORE_NAME" \
'[.tenants[] | select(.id == $tenant and .displayName == $name)] | length == 1' \
< "$AUTH9_WORK_DIR/tenant-restored-list-api.json" >/dev/null
printf '%s\n' \
"procedure=$AUTH9_TENANT_FORCING_PROCEDURE_ID" \
"tenant=$AUTH9_DISPOSABLE_TENANT" \
"approval=$AUTH9_DISPOSABLE_TENANT_APPROVAL" \
"before_display_name=$AUTH9_TENANT_BEFORE_DISPLAY_NAME" \
"before_version=$AUTH9_TENANT_BEFORE_VERSION" \
"update_version=$AUTH9_TENANT_UPDATE_VERSION" \
"restore_version=$AUTH9_TENANT_RESTORE_VERSION" \
"replica_container=$AUTH9_TENANT_REPLICA_CONTAINER" \
"replica_database=$AUTH9_TENANT_REPLICA_DATABASE" \
"replica_consumer=$AUTH9_TENANT_REPLICA_CONSUMER" \
"update_correlation=$AUTH9_TENANT_UPDATE_CORRELATION_ID" \
"restore_correlation=$AUTH9_TENANT_RESTORE_CORRELATION_ID" \
> "$AUTH9_WORK_DIR/tenant-forcing-record.txt"
trap - EXIT INT TERM
unset AUTH9_TENANT_ADMIN_TOKEN
The two API calls intentionally leave two immutable audit/outbox events and two monotonic versions. Those are the proof; the restored business value is exact. If an automatic restore fails, do not release the source fence or continue the window. There is still no tenant-delete endpoint, and this ruling is not authorization to add one.
Run the deferred AUTH-6 auth-critical Console gate against the exact deployed source, then prove AUTH-7 both at Authority’s authenticated surface and in Platform’s live registry. The four-persona script performs real login journeys with disposable users and owns their cleanup. AUTH9_OPS_HEALTH_TOKEN must carry ops.health; keep it only in memory.
pwsh tools/scripts/run-console-four-role-live.ps1 \
-RunId "$AUTH9_AUTH6_RUN_ID" \
-EvidenceDate "$AUTH9_EVIDENCE_DATE" \
-ExpectedCommit "$AUTH9_APPROVED_SOURCE_SHA" \
| tee "$AUTH9_WORK_DIR/auth6-four-persona-live.txt"
read -r -s -p 'AUTH-9 ops.health token: ' AUTH9_OPS_HEALTH_TOKEN; printf '\n'
test -n "$AUTH9_OPS_HEALTH_TOKEN"
authority_doctor_code="$(curl -sk \
-o "$AUTH9_WORK_DIR/authority-doctor-checks.json" -w '%{http_code}' \
-H "Authorization: Bearer $AUTH9_OPS_HEALTH_TOKEN" \
https://authority.stella-ops.local/doctor/authority/checks)"
[[ "$authority_doctor_code" == 200 ]]
jq -e '
.service == "authority" and .contractVersion == "doctor-check/v1" and
([.checks[].checkId] | sort) == [
"doctor.authority.break-glass.unreconciled",
"doctor.authority.issuer-fold.delete-guard",
"doctor.authority.issuer-fold.staging",
"doctor.authority.tenants-catalog.producer",
"doctor.authority.updated-at-helper.qualification",
"doctor.standard.db.connection",
"doctor.standard.db.migration-status",
"doctor.standard.db.size-budget"
]' "$AUTH9_WORK_DIR/authority-doctor-checks.json" >/dev/null
platform_doctor_code="$(curl -sk \
-o "$AUTH9_WORK_DIR/platform-authority-doctor-capabilities.json" -w '%{http_code}' \
-H "Authorization: Bearer $AUTH9_OPS_HEALTH_TOKEN" \
https://platform.stella-ops.local/api/v1/platform/doctor/capabilities)"
[[ "$platform_doctor_code" == 200 ]]
jq -e '[.capabilities[] | select(.serviceName == "authority")] as $authority |
($authority | length) == 8 and
($authority | all(
.contractVersion == "doctor-check/v1" and
.endpoint == "http://authority.stella-ops.local/doctor/authority/checks" and
.stale == false))' \
"$AUTH9_WORK_DIR/platform-authority-doctor-capabilities.json" >/dev/null
# COUNT AMENDED 7 -> 8, 2026-09-08 (window-4 pre-flight). AUTH-11
# (`f489352e53`) added `doctor.authority.break-glass.unreconciled` to
# `AuthorityDoctorChecks`, and this gate still named the pre-AUTH-11 set of
# seven. Measured on the window-4 image, which carries AUTH-11:
# `GET /doctor/authority/checks` answers 200 with EIGHT checks, all healthy.
# Left at seven, all three assertions here would have refused a correct estate
# in the middle of the outage -- the checkId list, the capability count and the
# registration log line. The extra check is an addition, not a replacement, so
# the other seven ids are unchanged.
docker logs --since "$AUTH9_AUTHORITY_RECREATE_SINCE" stellaops-authority 2>&1 \
| grep -F 'Doctor capability catalog registered: authority, 8 check(s).' \
| tee "$AUTH9_WORK_DIR/authority-doctor-registration.log"
unset AUTH9_OPS_HEALTH_TOKEN
Once that gate has a real approved value and its forcing pair passes, run the bounded soak. Every active caller must remain healthy, the folded surface must remain non-404, no process may wait behind the legacy source lock, and no permission/RLS error may appear since the soak start.
AUTH9_SOAK_SINCE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
mapfile -t AUTH9_SOAK_CONTAINERS < <(
printf '%s\n' stellaops-authority stellaops-router-gateway "${AUTH9_RUNNING_CALLERS[@]}" \
| awk '!seen[$0]++'
)
: > "$AUTH9_WORK_DIR/runtime-health-loop.tsv"
for _ in $(seq 1 "$((AUTH9_SOAK_MINUTES * 2))"); do
code="$(curl -sk -o /dev/null -w '%{http_code}' \
-H "Authorization: Bearer $AUTH9_ISSUER_TOKEN" \
-H "X-StellaOps-TenantId: $AUTH9_CANARY_TENANT" \
https://stella-ops.local/issuer-directory/issuers)"
[[ "$code" == 200 ]]
for container in "${AUTH9_SOAK_CONTAINERS[@]}"; do
runtime_state="$(docker inspect "$container" \
--format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}')"
[[ "$runtime_state" == healthy || "$runtime_state" == running ]] || {
echo "NO-GO: runtime-discovered caller/router is not healthy: $container=$runtime_state" >&2
exit 1
}
printf '%s\t%s\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$container" "$runtime_state" \
>> "$AUTH9_WORK_DIR/runtime-health-loop.tsv"
done
blocked="$(docker exec stellaops-postgres psql -XqAt -U stellaops -d postgres -c \
"SELECT count(*) FROM pg_stat_activity WHERE wait_event_type='Lock' AND pid <> (SELECT pid FROM pg_stat_activity WHERE application_name='$AUTH9_SOURCE_FENCE_APP');")"
[[ "$blocked" == 0 ]] || { echo "NO-GO: $blocked session(s) attempted a fenced legacy write" >&2; exit 1; }
sleep 30
done
while IFS= read -r container; do
docker logs --since "$AUTH9_SOAK_SINCE" "$container" 2>&1 \
> "$AUTH9_WORK_DIR/${container}.soak.log"
if grep -Eqi 'permission denied|insufficient_privilege|SQLSTATE[ =:]*(42501)|row.level security' \
"$AUTH9_WORK_DIR/${container}.soak.log"; then
echo "NO-GO: permission/RLS error during soak: $container" >&2; exit 1
fi
done < <(printf '%s\n' "${AUTH9_SOAK_CONTAINERS[@]}")
docker exec stellaops-postgres psql -XqAt -U stellaops -d postgres -c \
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name='$AUTH9_SOURCE_FENCE_APP';" \
| grep -qx t
[[ "$(docker exec stellaops-postgres psql -XqAt -U stellaops -d postgres -c \
"SELECT count(*) FROM pg_stat_activity WHERE application_name='$AUTH9_SOURCE_FENCE_APP';")" == 0 ]]
unset AUTH9_ISSUER_TOKEN AUTH9_LOW_PRIVILEGE_TOKEN AUTH9_TENANT_ADMIN_TOKEN
Record the completed snapshot record, before/after parity files, image ID, inventory, route check, forcing responses, named replica proof, health loop and zero-match log scan. Only then may AUTH-9 and the ownership matrix flip.
10. Exact rollback entry and exit
Keep the source lock until the legacy path is fully restored. If the issuer canary is active, delete it through its API first; never SQL-clean it.
AUTH9_ROLLBACK_SINCE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
# Re-establish the hard ingress/caller fence before changing the route back.
while IFS= read -r container; do
docker inspect "$container" >/dev/null
docker stop "$container" >/dev/null
done < "$AUTH9_WORK_DIR/writer-fence-stopped.txt"
docker start stellaops-issuer-directory >/dev/null
set +e
route_rollback_precheck="$(pwsh tools/scripts/auth5-swap-issuer-directory-routes.ps1 -Rollback -Check 2>&1)"
route_rollback_precheck_status=$?
set -e
printf '%s\n' "$route_rollback_precheck" \
| tee "$AUTH9_WORK_DIR/route-rollback-precheck.txt"
case "$route_rollback_precheck_status" in
0)
grep -Fq 'already restored to the legacy target' <<< "$route_rollback_precheck"
;;
1)
[[ "$(grep -c ' PENDING /issuer-directory' <<< "$route_rollback_precheck")" == 2 ]]
grep -Fq -- '-Check: 2 config(s) would change. Nothing written.' \
<<< "$route_rollback_precheck"
pwsh tools/scripts/auth5-swap-issuer-directory-routes.ps1 -Rollback
pwsh tools/scripts/auth5-swap-issuer-directory-routes.ps1 -Rollback -Check \
| tee "$AUTH9_WORK_DIR/route-rollback-applied.txt"
;;
*)
echo "NO-GO: route rollback precheck exited $route_rollback_precheck_status" >&2
exit 1
;;
esac
# Restore/unset this value in the protected env file before recreating callers:
# AUTHORITY_ISSUER_DIRECTORY_DIRECT_BASE_ADDRESS=http://issuerdirectory.stella-ops.local
for container in "${AUTH9_RUNNING_CALLERS[@]}"; do
auth9_compose_args_for "$container"
case "$container" in
stellaops-platform) env_key='STELLAOPS_ISSUERDIRECTORY_URL' ;;
*) env_key='IssuerDirectory__Client__BaseAddress' ;;
esac
rendered_target="$(docker "${AUTH9_COMPOSE_ARGS[@]}" config --format json \
| jq -er --arg service "$AUTH9_COMPOSE_SERVICE" --arg key "$env_key" \
'.services[$service].environment[$key]')"
[[ "$rendered_target" == 'http://issuerdirectory.stella-ops.local' ]] || {
echo "NO-GO: rollback renders $container $env_key=$rendered_target" >&2; exit 1;
}
docker "${AUTH9_COMPOSE_ARGS[@]}" up -d --force-recreate --no-deps "$AUTH9_COMPOSE_SERVICE"
docker inspect "$container" --format '{{range .Config.Env}}{{println .}}{{end}}' \
| grep -Fx "${env_key}=http://issuerdirectory.stella-ops.local"
done
# After restoring the protected pre-window stellaops connection and setting
# AUTHORITY_ISSUER_DIRECTORY_FOLD_ENABLED=false in the env file, recreate
# Authority from its exact PRE-WINDOW label chain. That chain already carries the
# recorded rollback image pin; do not append the AUTH-9 fold overlay/final pin.
auth9_compose_args_for stellaops-authority
docker "${AUTH9_COMPOSE_ARGS[@]}" up -d --force-recreate --no-deps authority
prewindow_authority_image_id="$(awk -F '\t' '$1 == "stellaops-authority" { print $5 }' \
"$AUTH9_WORK_DIR/container-inventory.tsv")"
[[ "$(docker inspect stellaops-authority --format '{{.Image}}')" == "$prewindow_authority_image_id" ]]
for _ in $(seq 1 60); do
[[ "$(docker inspect stellaops-authority --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}')" == healthy ]] && break
sleep 1
done
[[ "$(docker inspect stellaops-authority --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}')" == healthy ]]
rollback_fold_env="$(docker inspect stellaops-authority --format '{{range .Config.Env}}{{println .}}{{end}}' \
| grep '^ISSUERDIRECTORY__FOLD__ENABLED=' || true)"
[[ -z "$rollback_fold_env" || "$rollback_fold_env" == 'ISSUERDIRECTORY__FOLD__ENABLED=false' ]]
rollback_direct_code="$(docker exec stellaops-authority bash -lc \
"exec 3<>/dev/tcp/127.0.0.1/80; printf 'GET /issuer-directory/issuers HTTP/1.1\\r\\nHost: authority.stella-ops.local\\r\\nConnection: close\\r\\n\\r\\n' >&3; IFS=' ' read -r _ code _ <&3; printf '%s' \"\$code\"")"
[[ "$rollback_direct_code" == 404 ]]
[[ "$(docker inspect stellaops-issuer-directory --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}')" == healthy ]]
auth9_issuer_fingerprints stellaops_platform "$AUTH9_WORK_DIR/issuer-source.rollback.tsv"
diff -u "$AUTH9_WORK_DIR/issuer-source.after.tsv" "$AUTH9_WORK_DIR/issuer-source.rollback.tsv"
# Run "Owner-role rollback" below now, while no process uses authority_admin.
# Only after it passes may the source table fence and external ingress reopen.
: "${AUTH9_OWNER_ROLLBACK_STATE:?set VERIFIED only after Owner-role rollback and its preflight pass}"
[[ "$AUTH9_OWNER_ROLLBACK_STATE" == VERIFIED ]]
mapfile -t AUTH9_ROLLBACK_SCAN_CONTAINERS < <(
printf '%s\n' stellaops-authority stellaops-router-gateway "${AUTH9_RUNNING_CALLERS[@]}" \
| awk '!seen[$0]++'
)
for container in "${AUTH9_ROLLBACK_SCAN_CONTAINERS[@]}"; do
docker logs --since "$AUTH9_ROLLBACK_SINCE" "$container" 2>&1 \
> "$AUTH9_WORK_DIR/${container}.rollback.log"
if grep -Eqi 'permission denied|insufficient_privilege|SQLSTATE[ =:]*(42501)|row.level security' \
"$AUTH9_WORK_DIR/${container}.rollback.log"; then
echo "NO-GO: permission/RLS error during rollback: $container" >&2
exit 1
fi
done
docker exec stellaops-postgres psql -XqAt -U stellaops -d postgres -c \
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE application_name='$AUTH9_SOURCE_FENCE_APP';" \
| grep -qx t
docker start stellaops-router-gateway >/dev/null
unset AUTH9_ISSUER_TOKEN AUTH9_LOW_PRIVILEGE_TOKEN AUTH9_TENANT_ADMIN_TOKEN
Rollback exit criteria: route script says rollback already applied; Authority fold is false; every recreated runtime caller renders the legacy target; legacy container is healthy; source fingerprints still equal their pre-window values; the owner-role preflight reproduces the inert role; and the same permission-error scan is clean from rollback start. The target copy/audit may remain inert for a retry; do not copy target-only audit rows back to Platform and do not invoke the cluster snapshot restore as a routine rollback.
Direct-consumer disposition (source-pinned, not inferred from address count)
The old shorthand said “six SDK projects, three compose addresses, therefore three other consumers.” That subtraction was wrong: Platform is one of the three addressed services but does not reference the SDK. The actual set is six SDK projects plus Platform’s direct HttpClient. The conformance guard IssuerDirectorySdkConsumers_AreExactlyTheDispositionedSet scans every production project reference so a seventh SDK project cannot appear silently.
| Project/caller | Window disposition |
|---|---|
StellaOps.Excititor.WebService | Active SDK host. Recreated with the shared direct-target variable set to Authority. |
StellaOps.Excititor.Worker | Active SDK host. Recreated with the same target. |
StellaOps.Excititor.Attestation | Library, not a process. Resolves the optional IIssuerDirectoryClient supplied by the two Excititor hosts above; it has no independent address. |
StellaOps.Policy.Engine | Compiled optional client, but canonical production compose supplies no IssuerDirectory:Client:BaseAddress; startup registers no client. The opt-in operator-signing proof overlay does configure it through the shared target, so a window using that overlay must recreate and verify this service too. |
StellaOps.ReleaseOrchestrator.WebApi | Compiled optional client, but canonical production compose supplies no base address; startup records the client as unconfigured. The opt-in operator-signing proof overlay configures it through the shared target, so include it in the recreate/proof set when that overlay is present in the recorded live chain. |
StellaOps.DeltaVerdict | Library only. No production source invokes AddScoringTrustProvider, so its required SDK client is not registered in a running host. The guard turns red if that changes. |
Platform OperatorProviderChange.IssuerDirectory | Active direct HttpClient, not one of the six SDK projects. Recreated with STELLAOPS_ISSUERDIRECTORY_URL resolved from the shared target. |
The canonical and supported legacy compose surfaces resolve the three always-addressed callers from the shared variable. The opt-in operator-signing proof overlay resolves its two additional optional SDK addresses from the same variable: AUTHORITY_ISSUER_DIRECTORY_DIRECT_BASE_ADDRESS. Its committed/default value is the legacy host, so landing this source change is behaviour-neutral. The only cutover value is http://authority.stella-ops.local; rollback restores the legacy value (or unsets the variable) before recreating callers.
Forcing rows are probes, not product data
Use deterministic identifiers containing the window id and UTC timestamp. The issuer probe is created through POST /issuer-directory/issuers, read through the same gateway, and removed through DELETE /issuer-directory/issuers/{id} with an admin token. It has no signing keys or trust overrides, so the evidence delete guard cannot legitimately retain it. The issuer.issuers row must be gone; the target-only issuer.audit create/delete records are retained under their normal P13 class as window evidence and are never copied back to the legacy DB.
The tenant probe follows a different, owner-approved rule because Authority maps no tenant delete endpoint and must not gain one for this window. Section 9 uses an existing disposable tenant and a paired PATCH/PATCH restoration. It captures the exact prior display name and row_version, proves the temporary value through Authority’s outbox, canonical Timeline audit and Policy’s catalog.replica.tenants checkpoint, then restores and re-proves the exact prior value. Never substitute SQL cleanup, tenant create/delete, suspend/resume, or a different consumer. The monotonic versions plus retained audit/outbox rows are evidence, not product-data residue.
Retire temporary snapshot copies after successful closure
After the canonical window and its receipt are complete, retain and verify the snapshot dumps outside the container before removing their in-container copies. A digest is not a recovery copy. Set AUTH9_DUMP_ARCHIVE to the retained directory and AUTH9_WINDOW_RECEIPT to the completed receipt. The following block checks every digest and removes only the verified files; it leaves unexpected files in place. The earlier issuer-only dump already has its own copy-then-remove step.
# Run after the canonical window has completed successfully.
# Set these to the retained backup directory and the completed window receipt.
(
set -euo pipefail
test -d "$AUTH9_DUMP_ARCHIVE"
test -f "$AUTH9_WINDOW_RECEIPT"
test -s "$AUTH9_WORK_DIR/dump-hashes.txt"
[[ "$AUTH9_WINDOW_ID" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]]
[[ "$AUTH9_DUMP_DIR" = "/tmp/auth9-$AUTH9_WINDOW_ID-snapshot" ]]
auth9_retire_manifest=$(cat "$AUTH9_WORK_DIR/dump-hashes.txt")
# Verify every recorded dump, including its retained copy, before deleting any file.
while read -r auth9_hash auth9_original; do
[[ "$auth9_hash" =~ ^[0-9a-f]{64}$ ]]
auth9_name=$(basename -- "$auth9_original")
[[ "$auth9_name" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*\.dump$ ]]
[[ "$auth9_original" = "$AUTH9_DUMP_DIR/$auth9_name" ]]
grep -F -- "$auth9_hash" "$AUTH9_WINDOW_RECEIPT" >/dev/null
auth9_retained_hash=$(sha256sum -- "$AUTH9_DUMP_ARCHIVE/$auth9_name" | awk '{print $1}')
[[ "$auth9_retained_hash" = "$auth9_hash" ]]
auth9_current_hash=$(MSYS_NO_PATHCONV=1 docker exec stellaops-postgres sha256sum "$auth9_original" | awk '{print $1}')
[[ "$auth9_current_hash" = "$auth9_hash" ]]
done <<< "$auth9_retire_manifest"
# Delete only the verified files. Unexpected directory contents are preserved.
while read -r auth9_hash auth9_original; do
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres rm -f -- "$auth9_original"
done <<< "$auth9_retire_manifest"
MSYS_NO_PATHCONV=1 docker exec stellaops-postgres rmdir -- "$AUTH9_DUMP_DIR"
)
Why there is no consolidated publish key (read before “fixing” it)
authority is a CONTINUATION host: the fold’s target IS the live StellaOps.Authority WebService. Registering a second publish-helper / ownership-manifest key over the same seed project would not describe a new service — it would re-count one source graph, the same reason the AdvisoryAI sibling gave for keeping the predecessors’ keys (160e268686). Build with the existing key:
devops/docker/build-service-publish.sh authority
Do NOT delete the issuer-directory-web publish key before AUTH-10. It still builds the container that is serving the /issuer-directory prefix today. Its entry is devops/docker/build-service-publish.sh:201.
Doctor-SDK prune trap — verified, not inherited
The Authority image now ships StellaOps.Doctor.Plugin.Abstractions.dll, which matches the image build’s *.Plugin.*.dll prune pattern by name. It is allowlisted in is_host_runtime_plugin_payload (devops/docker/build-service-publish.sh, the StellaOps.Doctor.Plugin.Abstractions.dll case) for every key — checked for this stage rather than assumed from the adoption guide. If authority ever crash-loops at boot with a FileNotFoundException for that assembly, that allowlist has regressed; do not work around it in the service.
The doctor surface, and why it is quiet until AUTH-9
AUTH-7 adopted doctor-check/v1 on this host: GET /doctor/authority/checks, gated on the ops.health capability, serving the three standard checks plus four family checks. The endpoint is additive and the registration is default-off, which is what makes adopting on the auth root safe.
Read the four family checks as the window’s own instrument panel:
| Check | Pre-window answer | What a Critical means |
|---|---|---|
doctor.authority.issuer-fold.staging | staged-empty, Info | migration 024 never converged here, or the folded home is PARTIAL — the parity gate cannot run |
doctor.authority.issuer-fold.delete-guard | enforced, Info | the operator-decision-signing-key BEFORE DELETE guard is missing or mis-bound — evidence-class rows are deletable and nothing errors (D-AUTH3-2) |
doctor.authority.updated-at-helper.qualification | Info, both helpers in their own schemas | the D-AUTH3-3 collision: an unqualified carry replaced authority.update_updated_at, which produces NO migration error |
doctor.authority.tenants-catalog.producer | Info, eventing.outbox present | AUTH-9’s reversible tenant display-name forcing pair will fail — the row_version trigger or the own-database outbox is missing |
staged-empty is the CORRECT pre-window answer — and on THIS estate it is the correct POST-window answer too. The issuer tables in stellaops_authority are empty on purpose: migration 024 is additively inert (AUTH-3) and the live issuer-directory container keeps serving from the shared platform database until AUTH-9 copies the rows. Measured read-only 2026-08-11, the source has nothing to copy: issuer in stellaops_platform holds 0 rows in all four domain tables. So an operator who waits for this check to flip to populated here will wait forever, and one who reads staged-empty as a fault will “fix” a non-problem. Confirm the source counts on the estate in front of you before deciding which of the two you are in.
Registration is OFF, and this family has two traps neither sibling has
Doctor__Registration__Enabled: "${AUTHORITY_DOCTOR_REGISTRATION_ENABLED:-false}"
Flip it only after this service’s Authority client holds platform:doctor:register. Then:
- Leave
Doctor__Registration__ClientIdUNSET. This host holds exactly ONEStellaOpsAuthClientOptions, and outside the local harness the Console workspace already owns it (plus the token cache) to mint the per-tenant console-workspace token. Setting a DIFFERENT client id fails the host closed rather than silently repointing the Console’s token mint at the doctor client — on the auth root that failure would have presented as “the whole Console is broken”, not as “doctor registration is misconfigured”. Grant the scope to the Console workspace client instead. - Authority is its own issuer. The registrar mints its client-credentials token from this service, and OpenIddict refuses token requests over plain HTTP (ID2083) — so
Doctor__Registration__Authoritymust be HTTPS andDoctor__Registration__TokenEndpointmust be explicit. It cannot deadlock startup (ServiceDoctorRegistraris aBackgroundService), so a wrong value shows up as a quiet retry loop in the logs, not as a failed boot.
Blank is not the same as unset. Compose passes ${VAR:-} through as an empty string; the host guards these with IsNullOrWhiteSpace, so a half-filled value refuses to start instead of 403-looping against Platform with an empty client id.
Bring the fold overlay up
The overlay is deliberately not in the default config_files list (the sibling precedent; DC-19: one routing change per program, in the window, never aliased early).
cd devops/compose
docker compose -f docker-compose.stella-infra.yml \
-f docker-compose.stella-services.yml \
-f docker-compose.authority.yml \
up -d --force-recreate --no-deps authority
A same-tag image does NOT swap on up -d— --force-recreate is mandatory, every time, not just the first. After the swap the service re-sends HELLO over Valkey; a gateway 503 "no instances available" immediately afterwards is fixed by one docker restart of the service.
config_files labels are heterogeneous in this estate: recreate each service with its OWN -f list, never with a list assembled for a different service.
Retiring the issuer-directory container — a window STEP, not a file
A compose overlay cannot remove a service. Nothing in docker-compose.authority.yml stops, aliases or shadows the live issuer-directory container, and it must not be made to. Work the canonical transaction at the top of this runbook exactly once: folded host/direct callers first, route retarget second, the smallest gateway forcing pair, then stop the legacy container by name. The older stop-then-retarget wording was wrong because it creates an avoidable 502/503 interval.
Do not run a second sequence from this section. At AUTH-10, after AUTH-9’s soak, delete the issuer-directory compose key and issuer-directory-web publish key; do not delete either during this window.
Verification trap carried from AUTH-5 (D-AUTH5B-13)
grep -c 'issuer-directory' returns 2 for appsettings.json and 3 for router-gateway-local.json while both hold exactly ONE route object and ONE ApprovedAuthPassthroughPrefixes entry. That asymmetry is the HEALTHY state. One hit per file is the auth-passthrough prefix (a real, separate, LOAD-BEARING occurrence, not a route); the deployed config carries one more only because it expands route objects across lines. An operator who “normalises” the count deletes the passthrough entry and silently strips the Bearer from the folded surface.
Exact database owner/grant/RLS procedure
AUTH-9 does not create a second role. It turns the baseline’s existing inert authority_admin into the one login/database owner and transfers only the four Authority-owned schemas: authority, authority_app, eventing, and issuer. Do not use REASSIGN OWNED BY stellaops: this transitional database still contains foreign legacy schemas, and that command would steal all of them.
The direct BYPASSRLS attribute is intentional and mandatory for this cutover. The current host does not set an app.current_tenant session variable and today runs as the stellaops superuser; membership in a bypass role does not inherit the attribute. authority_admin itself therefore remains NOSUPERUSER and BYPASSRLS. The live preflight measured 2026-08-22 is: database plus all four owned schemas owned by stellaops; authority_admin is NOLOGIN, NOSUPERUSER, BYPASSRLS, has zero table grants, and cannot connect to either stellaops_authority or stellaops_platform. Re-measure; do not inherit it.
1. Read-only preflight
Run this before taking the write fence. Any extra Authority-owned schema or any object already owned by a third role stops the procedure for reconciliation.
docker exec -i stellaops-postgres psql -X -v ON_ERROR_STOP=1 \
-U stellaops -d stellaops_authority <<'SQL'
SELECT current_database(), pg_get_userbyid(datdba)
FROM pg_database WHERE datname = current_database();
SELECT rolname, rolcanlogin, rolsuper, rolinherit, rolbypassrls
FROM pg_roles WHERE rolname = 'authority_admin';
SELECT nspname, pg_get_userbyid(nspowner)
FROM pg_namespace
WHERE nspname IN ('authority','authority_app','eventing','issuer')
ORDER BY nspname;
SELECT n.nspname, c.relkind, pg_get_userbyid(c.relowner), count(*)
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname IN ('authority','authority_app','eventing','issuer')
AND c.relkind IN ('r','p','S','v','m','f')
GROUP BY 1,2,3 ORDER BY 1,2,3;
SELECT n.nspname, p.prokind, pg_get_userbyid(p.proowner), count(*)
FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname IN ('authority','authority_app','eventing','issuer')
GROUP BY 1,2,3 ORDER BY 1,2,3;
SELECT datname, has_database_privilege('authority_admin', datname, 'CONNECT')
FROM pg_database
WHERE datname IN ('stellaops_authority','stellaops_platform')
ORDER BY datname;
DO $auth9_preflight$
BEGIN
IF (SELECT count(*) FROM pg_roles WHERE rolname = 'authority_admin') <> 1 THEN
RAISE EXCEPTION 'AUTH-9 preflight: expected exactly one authority_admin role';
END IF;
IF EXISTS (
SELECT 1 FROM pg_roles WHERE rolname = 'authority_admin'
AND (rolcanlogin OR rolsuper OR rolcreatedb OR rolcreaterole
OR NOT rolinherit OR NOT rolbypassrls)
) THEN
RAISE EXCEPTION 'AUTH-9 preflight: authority_admin attributes are not NOLOGIN NOSUPERUSER INHERIT BYPASSRLS';
END IF;
IF (SELECT pg_get_userbyid(datdba) FROM pg_database
WHERE datname = current_database()) IS DISTINCT FROM 'stellaops' THEN
RAISE EXCEPTION 'AUTH-9 preflight: database has an unexpected owner';
END IF;
IF (SELECT count(*) FROM pg_namespace
WHERE nspname IN ('authority','authority_app','eventing','issuer')) <> 4 THEN
RAISE EXCEPTION 'AUTH-9 preflight: one or more owned schemas are absent';
END IF;
IF EXISTS (
SELECT 1 FROM pg_namespace
WHERE nspname IN ('authority','authority_app','eventing','issuer')
AND pg_get_userbyid(nspowner) <> 'stellaops'
) THEN
RAISE EXCEPTION 'AUTH-9 preflight: target schema has an unexpected owner';
END IF;
IF EXISTS (
SELECT 1 FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname IN ('authority','authority_app','eventing','issuer')
AND c.relkind IN ('r','p','S','v','m','f')
AND pg_get_userbyid(c.relowner) <> 'stellaops'
) THEN
RAISE EXCEPTION 'AUTH-9 preflight: target relation has an unexpected owner';
END IF;
IF EXISTS (
SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname IN ('authority','authority_app','eventing','issuer')
AND pg_get_userbyid(p.proowner) <> 'stellaops'
) THEN
RAISE EXCEPTION 'AUTH-9 preflight: target routine has an unexpected owner';
END IF;
IF EXISTS (
SELECT 1 FROM pg_database
WHERE datname IN ('stellaops_authority','stellaops_platform')
AND has_database_privilege('authority_admin', datname, 'CONNECT')
) THEN
RAISE EXCEPTION 'AUTH-9 preflight: inert authority_admin unexpectedly has CONNECT';
END IF;
END
$auth9_preflight$;
SQL
Required pre-state: exactly one authority_admin row; rolcanlogin=false, rolsuper=false, rolbypassrls=true; both database privilege answers false; and every listed object owner is stellaops.
2. Atomic role, ownership and grants
Read a unique service password into the operator shell; never put it in argv, history, the sprint log, or set -x output. docker exec -e NAME forwards only the variable name, and psql’s \getenv keeps the value out of the SQL text.
read -r -s -p 'authority_admin password: ' AUTHORITY_OWNER_PASSWORD; printf '\n'
test -n "$AUTHORITY_OWNER_PASSWORD"
export AUTHORITY_OWNER_PASSWORD
docker exec -i -e AUTHORITY_OWNER_PASSWORD stellaops-postgres \
psql -X -v ON_ERROR_STOP=1 -U stellaops -d stellaops_authority <<'SQL'
\getenv authority_owner_password AUTHORITY_OWNER_PASSWORD
BEGIN;
DO $auth9$
BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'authority_admin') THEN
RAISE EXCEPTION 'authority_admin is absent; migration 001 is not converged';
END IF;
IF EXISTS (
SELECT 1 FROM pg_roles
WHERE rolname = 'authority_admin' AND (rolsuper OR NOT rolbypassrls)
) THEN
RAISE EXCEPTION 'authority_admin attributes drifted; expected NOSUPERUSER BYPASSRLS';
END IF;
END
$auth9$;
ALTER ROLE authority_admin
LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT BYPASSRLS
PASSWORD :'authority_owner_password';
ALTER DATABASE stellaops_authority OWNER TO authority_admin;
REVOKE CONNECT ON DATABASE stellaops_authority FROM PUBLIC;
GRANT CONNECT ON DATABASE stellaops_authority TO authority_admin;
DO $auth9$
DECLARE
obj record;
object_kind text;
BEGIN
FOR obj IN
SELECT n.nspname, c.relname, c.relkind
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname IN ('authority','authority_app','eventing','issuer')
AND c.relkind IN ('r','p','S','v','m','f')
ORDER BY n.nspname, c.relname
LOOP
object_kind := CASE obj.relkind
WHEN 'r' THEN 'TABLE' WHEN 'p' THEN 'TABLE'
WHEN 'S' THEN 'SEQUENCE' WHEN 'v' THEN 'VIEW'
WHEN 'm' THEN 'MATERIALIZED VIEW' WHEN 'f' THEN 'FOREIGN TABLE'
END;
EXECUTE format('ALTER %s %I.%I OWNER TO authority_admin',
object_kind, obj.nspname, obj.relname);
END LOOP;
FOR obj IN
SELECT n.nspname, p.proname, p.prokind,
pg_get_function_identity_arguments(p.oid) AS args
FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname IN ('authority','authority_app','eventing','issuer')
ORDER BY n.nspname, p.proname, args
LOOP
object_kind := CASE obj.prokind
WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE'
ELSE 'FUNCTION'
END;
EXECUTE format('ALTER %s %I.%I(%s) OWNER TO authority_admin',
object_kind, obj.nspname, obj.proname, obj.args);
END LOOP;
END
$auth9$;
ALTER SCHEMA authority OWNER TO authority_admin;
ALTER SCHEMA authority_app OWNER TO authority_admin;
ALTER SCHEMA eventing OWNER TO authority_admin;
ALTER SCHEMA issuer OWNER TO authority_admin;
GRANT USAGE, CREATE ON SCHEMA authority, authority_app, eventing, issuer
TO authority_admin;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA authority, authority_app, eventing, issuer
TO authority_admin;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA authority, authority_app, eventing, issuer
TO authority_admin;
GRANT EXECUTE ON ALL ROUTINES IN SCHEMA authority, authority_app, eventing, issuer
TO authority_admin;
DO $auth9_post_transfer$
BEGIN
IF (SELECT pg_get_userbyid(datdba) FROM pg_database
WHERE datname = current_database()) IS DISTINCT FROM 'authority_admin' THEN
RAISE EXCEPTION 'AUTH-9 post-transfer: database owner is not authority_admin';
END IF;
IF EXISTS (
SELECT 1 FROM pg_namespace
WHERE nspname IN ('authority','authority_app','eventing','issuer')
AND pg_get_userbyid(nspowner) <> 'authority_admin'
) THEN
RAISE EXCEPTION 'AUTH-9 post-transfer: target schema has an unexpected owner';
END IF;
IF EXISTS (
SELECT 1 FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname IN ('authority','authority_app','eventing','issuer')
AND c.relkind IN ('r','p','S','v','m','f')
AND pg_get_userbyid(c.relowner) <> 'authority_admin'
) THEN
RAISE EXCEPTION 'AUTH-9 post-transfer: target relation has an unexpected owner';
END IF;
IF EXISTS (
SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname IN ('authority','authority_app','eventing','issuer')
AND pg_get_userbyid(p.proowner) <> 'authority_admin'
) THEN
RAISE EXCEPTION 'AUTH-9 post-transfer: target routine has an unexpected owner';
END IF;
END
$auth9_post_transfer$;
COMMIT;
SQL
Change only the git-ignored/protected STELLAOPS_POSTGRES_AUTHORITY_CONNECTION value in devops/compose/.env to Host=db.stella-ops.local;Database=stellaops_authority;Username=authority_admin plus the same in-memory password and the existing port/pool options. Never record the rendered password. Both Authority storage and folded issuer storage must render from this one canonical value.
3. Read-only proof before recreate
# Own database and RLS-positive control: must print four fields and a tenant count.
export PGPASSWORD="$AUTHORITY_OWNER_PASSWORD"
docker exec -e PGPASSWORD stellaops-postgres \
psql -X -v ON_ERROR_STOP=1 -U authority_admin -d stellaops_authority -At \
-c "SELECT current_user,(SELECT count(*) FROM authority.tenants),
(SELECT count(*) FROM issuer.issuers),
(SELECT count(*) FROM eventing.outbox);"
# Same-database foreign schema and sibling database must both REFUSE access
# with SQLSTATE 42501. A typo, missing object, or network failure is not proof.
set +e
foreign_schema_denial="$(docker exec -e PGPASSWORD stellaops-postgres \
psql -X -v ON_ERROR_STOP=1 -v VERBOSITY=verbose \
-U authority_admin -d stellaops_authority \
-c 'SELECT 1 FROM shared.tenants LIMIT 1' 2>&1)"
foreign_schema_status=$?
sibling_database_denial="$(docker exec -e PGPASSWORD stellaops-postgres \
psql -X -v ON_ERROR_STOP=1 -v VERBOSITY=verbose \
-U authority_admin -d stellaops_platform -c 'SELECT 1' 2>&1)"
sibling_database_status=$?
set -e
printf '%s\n' "$foreign_schema_denial" \
> "$AUTH9_WORK_DIR/authority-owner-foreign-schema-denial.txt"
printf '%s\n' "$sibling_database_denial" \
> "$AUTH9_WORK_DIR/authority-owner-sibling-database-denial.txt"
[[ "$foreign_schema_status" != 0 && "$sibling_database_status" != 0 ]]
grep -Eq '(^|[[:space:]])42501([:[:space:]]|$)' \
"$AUTH9_WORK_DIR/authority-owner-foreign-schema-denial.txt"
# The FOREIGN-SCHEMA refusal is a server-side query error and does carry SQLSTATE 42501
# (asserted above). The SIBLING-DATABASE refusal does NOT, and demanding one there fails
# the gate on a correct result: a refused CONNECT is a libpq CONNECTION failure, which
# psql prints as `FATAL: permission denied for database ...` with no SQLSTATE even under
# VERBOSITY=verbose. Measured 2026-09-07 in the executed window (D-AUTH9-35). Assert the
# refusal text AND the catalog state instead - the catalog is the stronger proof, because
# it cannot be satisfied by a typo, a missing object or a network error, which is exactly
# what the original SQLSTATE demand was guarding against.
grep -Eq 'permission denied for database' \
"$AUTH9_WORK_DIR/authority-owner-sibling-database-denial.txt"
# Scoped to the SIBLING SERVICE databases, which is what CoC 8.2 actually forbids reaching.
# Do not widen it to every database: `postgres` and `template1` keep PUBLIC CONNECT by
# design, so an unscoped count is never 0 and the gate would fail permanently.
# Red-proved 2026-09-07: 0 inert, 1 after GRANT CONNECT on stellaops_policy, 0 after REVOKE.
[[ "$(docker exec stellaops-postgres psql -XqAt -U stellaops -d postgres -c "SELECT count(*) FROM pg_database WHERE datname LIKE 'stellaops%' AND datname <> 'stellaops_authority' AND has_database_privilege('authority_admin', datname, 'CONNECT');")" == 0 ]]
unset PGPASSWORD
unset AUTHORITY_OWNER_PASSWORD
The first command proves direct-login BYPASSRLS is effective without a tenant session variable. The two negative controls must fail with permission/connection denied, not with a misspelled object or network error. Also assert zero owned objects remain under stellaops in the four schemas before recreating Authority.
4. Owner-role rollback
Route/fold rollback comes first (companion runbook): restore the legacy container and route, set AUTHORITY_ISSUER_DIRECTORY_DIRECT_BASE_ADDRESS back to http://issuerdirectory.stella-ops.local (or unset it), repoint/recreate the three current direct consumers plus Policy/ReleaseOrchestrator when the recorded chain contains the opt-in proof overlay, then recreate Authority with the old protected stellaops connection and fold off. Only when no process uses authority_admin may Database Operations reverse ownership:
docker exec -i stellaops-postgres psql -X -v ON_ERROR_STOP=1 \
-U stellaops -d stellaops_authority <<'SQL'
BEGIN;
DO $auth9$
DECLARE
obj record;
object_kind text;
BEGIN
FOR obj IN
SELECT n.nspname, c.relname, c.relkind
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname IN ('authority','authority_app','eventing','issuer')
AND c.relkind IN ('r','p','S','v','m','f')
ORDER BY n.nspname, c.relname
LOOP
object_kind := CASE obj.relkind
WHEN 'r' THEN 'TABLE' WHEN 'p' THEN 'TABLE'
WHEN 'S' THEN 'SEQUENCE' WHEN 'v' THEN 'VIEW'
WHEN 'm' THEN 'MATERIALIZED VIEW' WHEN 'f' THEN 'FOREIGN TABLE'
END;
EXECUTE format('ALTER %s %I.%I OWNER TO stellaops',
object_kind, obj.nspname, obj.relname);
END LOOP;
FOR obj IN
SELECT n.nspname, p.proname, p.prokind,
pg_get_function_identity_arguments(p.oid) AS args
FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname IN ('authority','authority_app','eventing','issuer')
ORDER BY n.nspname, p.proname, args
LOOP
object_kind := CASE obj.prokind
WHEN 'p' THEN 'PROCEDURE' WHEN 'a' THEN 'AGGREGATE'
ELSE 'FUNCTION'
END;
EXECUTE format('ALTER %s %I.%I(%s) OWNER TO stellaops',
object_kind, obj.nspname, obj.proname, obj.args);
END LOOP;
END
$auth9$;
ALTER SCHEMA authority OWNER TO stellaops;
ALTER SCHEMA authority_app OWNER TO stellaops;
ALTER SCHEMA eventing OWNER TO stellaops;
ALTER SCHEMA issuer OWNER TO stellaops;
ALTER DATABASE stellaops_authority OWNER TO stellaops;
REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA authority, authority_app, eventing, issuer
FROM authority_admin;
REVOKE ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA authority, authority_app, eventing, issuer
FROM authority_admin;
REVOKE EXECUTE ON ALL ROUTINES IN SCHEMA authority, authority_app, eventing, issuer
FROM authority_admin;
REVOKE ALL PRIVILEGES ON SCHEMA authority, authority_app, eventing, issuer
FROM authority_admin;
REVOKE CONNECT ON DATABASE stellaops_authority FROM authority_admin;
ALTER ROLE authority_admin
NOLOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT BYPASSRLS PASSWORD NULL;
COMMIT;
SQL
unset AUTHORITY_OWNER_PASSWORD
Re-run the preflight: it must reproduce the recorded pre-state. Public CONNECT is deliberately not restored; it was already revoked before AUTH-9. Never drop the role: migration 001 owns its existence and expects the inert role to remain.
The refusal mechanism is guarded by AuthorityDoctorAdoptionTests.OwnerRoleGrants_RefuseCrossOwnerAccess_AndTheBaselinesOwnRoleIsInert: a real LOGIN role reaches its own schemas and gets insufficient_privilege for a foreign owner’s reads and writes, so the negative proof cannot pass merely because the test role is unusable.
Stale configuration this fold leaves behind
devops/etc/issuer-directory/issuer-directory.yaml carries a committed default postgresConnectionString pointing at Database=stellaops_platform with schemaName: issuer. Compose overrides it today, and the overlay overrides it after the fold — but it is what the folded surface would silently fall back to if the override were ever dropped. Treat it as a hazard for AUTH-10 to delete, not as a fallback to rely on. (The same file’s seedCsafPublishers: true is already overridden to false by compose.)
What AUTH-9 still owes after this overlay is healthy
the endpoint mapping (D-AUTH7-5)— CLOSED 2026-08-11 at AUTH-5. The host maps the folded groups behindISSUERDIRECTORY__FOLD__ENABLED(default false). What the window owes instead is flipping that switch as part of the single fenced transaction at the top of this runbook;- execute (do not redesign) the
authority_adminowner/grant/RLS procedure above, switch the protected canonical connection, and soak on both positive own-schema access and negative shared/sibling access; - copy the
issuerschema and record row-count + spot-check parity; the own side of that gate is readable straight fromdoctor.authority.issuer-fold.staging. Measured read-only on this estate 2026-08-11: the SOURCE is EMPTY —issuerinstellaops_platformholds 0 rows inissuers,issuer_keys,trust_overridesandaudit, so the “≈352 kB” in the sprint text is index and catalogue overhead, not data. The copy is a no-op and the parity gate is 0 = 0, which also meansdoctor.authority.issuer-fold.stagingcorrectly staysstaged-emptyAFTER the window on this estate. Re-measure on any other estate; do not inherit this number; the live scheme probe (F-AUTH5-1)— ANSWERED 2026-08-11, and the premise was wrong: NEITHER surface is stale. The compose sweep that produced the finding could only ever have found half the answer, because Authority’shttps://…:443listener is added in CODE, not compose:TryAddStellaOpsLocalBinding("authority")(StellaOpsLocalHostnameExtensions.cs:61) binds BOTHhttps://{service}.stella-ops.local:443andhttp://…:80on top of whateverASPNETCORE_URLSasked for, presenting the Kestrel default certificate that compose DOES configure for this container. Measured live, read-only: the authority container listens on 80, 443 and 8440, and a request through the gateway tohttps://stella-ops.local/connect/tokenreaches Authority’s OpenIddict token endpoint and gets an application-level 400 — i.e. the https upstream serves. The seven https routes are correct as they stand and no reconciliation is owed. Read-across: never conclude “nothing serves this scheme” from a compose/Dockerfile sweep in this estate — every host callingTryAddStellaOpsLocalBindingadds listeners no compose file mentions;- the scratch-stack authorized/unauthorized forcing pair (AUTH-5) and the four-persona live gate (AUTH-6), both explicitly not claimed by their stages;
- execute
AUTH9-TENANT-DISPLAY-NAME-REVERSIBLE-V1against the owner-designated disposable tenant, including exact prior-value restoration, two canonical Timeline audit records and Policy replica/checkpoint proof; - flip
AUTHORITY_DOCTOR_REGISTRATION_ENABLEDafter the scope grant, and confirmregistered: N check(s)in the log plus the capability rows in the Platform registry; - apply and prove the P13 headers for the 25-declaration
authoritylineage: 22 tables are extant on a fresh chain after001drops three deprecated local audit tables. Forward026classifies all missing extant tables and any of those three retained on an upgraded estate;S050prevents appliedS001from stripping theclient_tenantsheader. Migration 024 separately carries headers on its own fiveissuertables.
