Runbook — Kestrel TLS certificate rotation

Audience: DevOps / Live-Deploy operators rotating the shared Kestrel server certificate in a local/dev Stella Ops estate or installing an operator-issued certificate in a non-development deployment.

Change class: scheduled, estate-wide TLS maintenance. This runbook is preparation, not approval to rotate a live estate. Obtain an explicit window before changing a mounted PFX, its password, its trust bundle, or any running container.

Related: live-deploy-operator.md, deployment-prerequisites.md, and SPRINT_20260821_002.

What changes

Most HTTPS roles load a password-protected PFX from /app/etc/certs; Authority may use /app/etc/authority/keys. Compose supplies the password through KESTREL_CERT_PASSWORD and can override the mounted directory and filenames through STELLAOPS_CERT_VOLUME, KESTREL_CERT_PATH, and KESTREL_AUTHORITY_CERT_PATH. Internal TLS clients mount one of two distinct source bundles: devops/compose/combined-ca-bundle.crt or devops/compose/gateway-ca-bundle.crt (or an operator-supplied equivalent). Both populations must trust the same current and next Kestrel chain.

Changing files on the host does not reload certificates already held in running processes. The window therefore uses dual trust: expand both bundles to old+new, recreate trust consumers, swap the PFX/password and recreate every Kestrel password/path holder, then contract both bundles to new-only and recreate trust consumers again. Never expose a new leaf before its clients trust it, and never remove old trust while any process can still serve the old leaf.

Important helper blast radius. devops/compose/scripts/ensure-dev-certs.sh --force is a legacy all-dev-material operation: it regenerates the Scanner scan-attestation key, Release Orchestrator gate-decision key, ExportCenter NIS2 SoA key, Kestrel key/certificate/PFX, and the dev trust-bundle markers in both source bundles. It is not a Kestrel-only command. Any bare --force refuses before any write, even when the PFX is absent. The deliberate form is --force --acknowledge-kestrel-rotation-window, but use it only when the approved window covers every listed key family. The helper is not the live cutover orchestrator and does not perform the dual-trust/recreate phases. It also has no output-directory option: it writes the compose tree relative to its own location. Copying the helper into a complete synthetic compose/etc layout is suitable for an isolated test, not a supported way to stage a live candidate. For a Kestrel-only rotation, use the out-of-place procedure below.

1. Preflight and inventory

Confirm the approved start/end time, operator, rollback owner, and a maintenance mode that keeps release mutations out of the window. Do not overlap another deploy, image build, database move, or secret rotation.

Create a protected workspace. Repo-local tmp/ is suitable only for throwaway local/dev material; non-development private material belongs on operator-controlled encrypted storage.

REPO_ROOT="$(git rev-parse --show-toplevel)"
WINDOW_ID="$(date -u +%Y%m%dT%H%M%SZ)"
ROTATION_ROOT="${REPO_ROOT}/tmp/kestrel-rotation/${WINDOW_ID}"   # local/dev only
CERT_DIR="${REPO_ROOT}/devops/etc/authority/keys"
COMBINED_CA_BUNDLE="${REPO_ROOT}/devops/compose/combined-ca-bundle.crt"
GATEWAY_CA_BUNDLE="${REPO_ROOT}/devops/compose/gateway-ca-bundle.crt"
AUTHORITY_CA_PIN="${REPO_ROOT}/devops/compose/authority-ca.crt"
STAGED_COMBINED_CA_BUNDLE="${REPO_ROOT}/devops/release/bundle/config/combined-ca-bundle.crt"
STAGED_GATEWAY_CA_BUNDLE="${REPO_ROOT}/devops/release/bundle/config/gateway-ca-bundle.crt"
CA_BUNDLES=("${COMBINED_CA_BUNDLE}" "${GATEWAY_CA_BUNDLE}")
PINNING_FILES=("${CA_BUNDLES[@]}" "${AUTHORITY_CA_PIN}" \
  "${STAGED_COMBINED_CA_BUNDLE}" "${STAGED_GATEWAY_CA_BUNDLE}")
umask 077
mkdir -p "${ROTATION_ROOT}/candidate" "${ROTATION_ROOT}/backup" \
  "${ROTATION_ROOT}/inventory" "${ROTATION_ROOT}/pins"

Verify the current PFX by operation, then record its fingerprint, expiry, and file digests. Read the old password from its current secret-store version; do not put it on the command line or in retained output.

export OLD_KESTREL_CERT_PASSWORD="<read from the current secret version>"
openssl pkcs12 -in "${CERT_DIR}/kestrel-dev.pfx" \
  -passin env:OLD_KESTREL_CERT_PASSWORD -clcerts -nokeys \
  -out "${ROTATION_ROOT}/backup/old-leaf.crt"
openssl x509 -in "${ROTATION_ROOT}/backup/old-leaf.crt" \
  -noout -sha256 -fingerprint -serial -subject -issuer -dates \
  | tee "${ROTATION_ROOT}/inventory/old-certificate.txt"
sha256sum "${CERT_DIR}/kestrel-dev.pfx" "${PINNING_FILES[@]}" \
  > "${ROTATION_ROOT}/inventory/old-file-digests.sha256"

Detect trust-marker/current-leaf drift before generating or installing anything. Each source bundle may have either zero markers (the measured historical gateway shape) or exactly one complete managed pair. A partial pair, multiple starts, multiple ends, or out-of-order pair is a hard refusal. Set KESTREL_TRUST_MODE explicitly to legacy-self-signed-leaf for the historical self-signed CA:FALSE leaf-anchor shape, or ca-chain for a valid CA-rooted deployment. Never infer the mode from a successful or failed platform trust check.

marker_begin='# BEGIN stellaops-dev-kestrel-trust'
marker_end='# END stellaops-dev-kestrel-trust'
KESTREL_TRUST_MODE="${KESTREL_TRUST_MODE:?set legacy-self-signed-leaf or ca-chain}"
old_leaf_fp="$(openssl x509 -in "${ROTATION_ROOT}/backup/old-leaf.crt" \
  -noout -sha256 -fingerprint | sed 's/^.*=//;s/://g')"

old_subject="$(openssl x509 -in "${ROTATION_ROOT}/backup/old-leaf.crt" \
  -noout -subject -nameopt RFC2253 | sed 's/^subject=//')"
old_issuer="$(openssl x509 -in "${ROTATION_ROOT}/backup/old-leaf.crt" \
  -noout -issuer -nameopt RFC2253 | sed 's/^issuer=//')"

for bundle in "${CA_BUNDLES[@]}"; do
  begin_count="$(grep -cFx "${marker_begin}" "${bundle}" || true)"
  end_count="$(grep -cFx "${marker_end}" "${bundle}" || true)"
  case "${begin_count}:${end_count}" in
    0:0) marker_state=absent ;;
    1:1)
      begin_line="$(grep -nFx "${marker_begin}" "${bundle}" | cut -d: -f1)"
      end_line="$(grep -nFx "${marker_end}" "${bundle}" | cut -d: -f1)"
      [ "${begin_line}" -lt "${end_line}" ] || {
        echo "refusing: ${bundle} has an out-of-order marker pair" >&2; exit 1;
      }
      marker_state=present
      ;;
    *) echo "refusing: ${bundle} has partial/multiple markers (${begin_count}/${end_count})" >&2;
       exit 1 ;;
  esac

  # Inventory EVERY PEM certificate before deciding what the marker represents.
  # This file is the approved exact-DER removal/preservation set for phases 1/3.
  python3 - "${bundle}" \
    > "${ROTATION_ROOT}/inventory/$(basename "${bundle}").fingerprints.psv" <<'PY'

import base64, hashlib, pathlib, re, sys data = pathlib.Path(sys.argv[1]).read_bytes() pattern = re.compile(rb’-----BEGIN CERTIFICATE-----\s*([A-Za-z0-9+/=\r\n]+?)\s*-----END CERTIFICATE-----‘) matches = list(pattern.finditer(data)) if data.count(b’-----BEGIN CERTIFICATE-----‘) != len(matches) or data.count(b’-----END CERTIFICATE-----‘) != len(matches): raise SystemExit(‘refusing: partial or malformed PEM certificate’) for index, match in enumerate(matches, 1): der = base64.b64decode(re.sub(rb’\s’, b’‘, match.group(1)), validate=True) print(f’{index}|{hashlib.sha256(der).hexdigest().upper()}') PY if [ “${KESTREL_TRUST_MODE}” = ‘legacy-self-signed-leaf’ ]; then grep -qF “${old_leaf_fp}”
“${ROTATION_ROOT}/inventory/$(basename “${bundle}”).fingerprints.psv” || { echo “refusing: ${bundle} does not pin the exact current PFX leaf” >&2; exit 1; } fi printf ‘%s|%s|%s\n’ “${bundle}” “${marker_state}” “${old_leaf_fp}”
>> “${ROTATION_ROOT}/inventory/trust-marker-shapes.psv”

 # Record this implementation-specific result, but do not gate legacy
 # classification on it: Git/OpenSSL can accept the exact CA:FALSE leaf as
 # an explicit trust anchor while another Linux/.NET posture rejects it.
 set +e
 openssl verify -purpose sslserver -CAfile "${bundle}" \
   "${ROTATION_ROOT}/backup/old-leaf.crt" \
   > "${ROTATION_ROOT}/inventory/$(basename "${bundle}").openssl-verify.txt" 2>&1
 verify_status=$?
 set -e
 printf '%s|%s\n' "${bundle}" "${verify_status}" \
   >> "${ROTATION_ROOT}/inventory/old-verify-status.psv"

done

if [ “${KESTREL_TRUST_MODE}” = ‘legacy-self-signed-leaf’ ]; then [ “${old_subject}” = “${old_issuer}” ] || { echo ‘refusing: legacy leaf is not self-signed by subject/issuer’ >&2; exit 1; } openssl x509 -in “${ROTATION_ROOT}/backup/old-leaf.crt” -noout -text
| grep -q ‘CA:FALSE’ || { echo ‘refusing: legacy leaf is not CA:FALSE’ >&2; exit 1; } openssl x509 -in “${ROTATION_ROOT}/backup/old-leaf.crt” -purpose
| grep -q ‘SSL server : Yes’ || { echo ‘refusing: legacy leaf lacks serverAuth’ >&2; exit 1; } elif [ “${KESTREL_TRUST_MODE}” = ‘ca-chain’ ]; then for bundle in “${CA_BUNDLES[@]}”; do openssl verify -purpose sslserver -CAfile “${bundle}”
“${ROTATION_ROOT}/backup/old-leaf.crt” done else echo “refusing: unsupported KESTREL_TRUST_MODE=${KESTREL_TRUST_MODE}” >&2; exit 1 fi

authority_pin_fp=“$(openssl x509 -in “${AUTHORITY_CA_PIN}”
-noout -sha256 -fingerprint | sed ‘s/^.*=//;s/://g’)” printf ‘%s|%s\n’ “${AUTHORITY_CA_PIN}” “${authority_pin_fp}”
> “${ROTATION_ROOT}/inventory/authority-ca-pin.psv” if [ “${KESTREL_TRUST_MODE}” = ‘legacy-self-signed-leaf’ ]; then [ “${authority_pin_fp}” = “${old_leaf_fp}” ] || { echo ‘refusing: authority-ca.crt is not the exact current legacy leaf’ >&2; exit 1; } else openssl verify -purpose sslserver -CAfile “${AUTHORITY_CA_PIN}”
“${ROTATION_ROOT}/backup/old-leaf.crt” fi cmp -s “${COMBINED_CA_BUNDLE}” “${STAGED_COMBINED_CA_BUNDLE}” || { echo ‘refusing: staged combined bundle is stale’ >&2; exit 1; } cmp -s “${GATEWAY_CA_BUNDLE}” “${STAGED_GATEWAY_CA_BUNDLE}” || { echo ‘refusing: staged gateway bundle is stale’ >&2; exit 1; }


Zero markers are recorded drift, not silent success: phase 1 may normalize that bundle only after
the exact fingerprint inventory above is reviewed and the current served leaf is proven. Partial
or multiple markers, malformed PEM, a missing current-leaf fingerprint, or an unexplained
certificate still blocks the swap. `authority-ca.crt` and both release-bundle staged copies make
five tracked pinning files total; they must be in the same fingerprint inventory even though only
the two source bundles carry managed markers. Approve the exact old/stale fingerprint set for
phase-1 preservation and phase-3 removal; never append/delete by subject or CN.

5. Inventory two sets from the running containers, not from a remembered compose list:

- **trust consumers** mounting either source bundle; these recreate in phases 1 and 3; and
- **PFX consumers** carrying the Kestrel default-certificate password and/or path setting; these
  recreate in phase 2 whether they advertise HTTPS, HTTP, a worker loop, or no socket at all.

For both sets record service, project, current image ID, relevant mount/path, and the container's
own `com.docker.compose.project.config_files` value. Inspect environment **names**, not their
secret values. Never union heterogeneous config-file chains.

```bash
for container in $(docker ps --format '{{.Names}}'); do
  # Emit only environment NAMES plus the non-secret certificate path; never
  # retain Config.Env or the password value.
  env_names="$(docker inspect "${container}" --format \
    '{{range .Config.Env}}{{println (index (split . "=") 0)}}{{end}}')"
  cert_path="$(docker inspect "${container}" --format \
    '{{range .Config.Env}}{{ $kv := split . "=" }}{{if eq (index $kv 0) "Kestrel__Certificates__Default__Path"}}{{index $kv 1}}{{end}}{{end}}')"
  record="$(docker inspect "${container}" --format \
    '{{index .Config.Labels "com.docker.compose.service"}}|{{index .Config.Labels "com.docker.compose.project"}}|{{.Image}}|{{index .Config.Labels "com.docker.compose.project.config_files"}}')"
  mounts="$(docker inspect "${container}" --format \
    '{{range .Mounts}}{{println .Source "|" .Destination}}{{end}}')"

  if printf '%s\n' "${mounts}" \
    | grep -Eq '(combined-ca-bundle|gateway-ca-bundle)\.crt'; then
    printf '%s|%s\n' "${record}" "${container}" \
      >> "${ROTATION_ROOT}/inventory/trust-consumers.psv"
  fi

  has_password=0
  has_path=0
  printf '%s\n' "${env_names}" \
    | grep -qxF 'Kestrel__Certificates__Default__Password' && has_password=1
  printf '%s\n' "${env_names}" \
    | grep -qxF 'Kestrel__Certificates__Default__Path' && has_path=1
  if [ "${has_password}" -eq 1 ] || [ "${has_path}" -eq 1 ]; then
    printf '%s|%s|%s|%s|%s\n' \
      "${record}" "${container}" "${cert_path}" "${has_password}" "${has_path}" \
      >> "${ROTATION_ROOT}/inventory/pfx-consumers.psv"
  fi
done

A socket inventory remains useful endpoint evidence, but it never selects phase-2 targets: Stella Ops local binding loads the PFX for HTTP-advertising roles and workers too. Recreate every current non-retired row in pfx-consumers.psv. For every image ID in both inventories, docker image inspect <id> must succeed, and the recreate pin must resolve to that exact ID. If metadata was pruned, stop and complete the recovery procedure in the live-deploy playbook; a moved tag is not a rollback image.

Record the exact, ordered compose invocation for every row in both inventories. The com.docker.compose.project.config_files label records config files but does not preserve CLI --env-file arguments. Recover those arguments from the deployment record/launcher and write their ordered paths to ${ROTATION_ROOT}/inventory/env-files.psv; if they cannot be proven, stop. Make the normally implicit devops/compose/.env explicit in the replay. The SBOM service is a mandatory two-file case and must render/recreate in this order:
```
sbom_env_args=(
  --env-file "${REPO_ROOT}/devops/compose/.env"
  --env-file "${REPO_ROOT}/devops/compose/.env.database-moves"
)
# Populate sbom_config_args from the recorded config_files label, preserving order.
docker compose -p stellaops "${sbom_env_args[@]}" "${sbom_config_args[@]}" \
  config --format json
```
The second file overrides the first; reversing them silently restores stale database values. Other special launchers (for example the agent runtime) may have a different additional env file, so do not copy the SBOM list globally. Before any recreate, replay that target’s exact env-file list and config-file chain, add only its verified image pin, render, and compare the service’s image, mounts, networks, certificate path, and environment names with the running container. Keep the render protected and delete it after review because it contains expanded secrets.
Capture a before-state that can fail: container health, direct TLS reachability, the served leaf fingerprint/expiry, OIDC discovery, and one normal login or client-credentials token flow. Keep tokens and secrets out of the evidence; retain status, timestamps, issuer/audience, and hashes. Inventory exited containers separately. A retired/exited doctor-web is an accepted baseline disposition, not a health target and never a recreate candidate; fail if a generated override would materialise it as a new service.

2. Prepare the candidate out of place

Set a new, randomly generated password in a new secret-store version. Keep the old version readable for rollback until the window is closed.

Local/dev ephemeral-CA candidate

Generate an ephemeral CA:TRUE local root, then issue a CA:FALSE, serverAuth leaf outside the mounted directory. Trust the root, not the leaf. A self-signed CA:FALSE leaf copied into a CA bundle is not a standards-valid CA chain even though some OpenSSL builds accept that exact leaf as an explicit trust anchor. Out-of-place generation also avoids the auxiliary-key blast radius of the helper’s --force.

export NEW_KESTREL_CERT_PASSWORD="<new random value from the dev secret source>"
cat > "${ROTATION_ROOT}/candidate/ca.cnf" <<'CFG'
[req]
distinguished_name = dn
x509_extensions = v3_ca
prompt = no
[dn]
CN = Stella Ops Local Dev Root CA
[v3_ca]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical,CA:TRUE,pathlen:0
keyUsage = critical,keyCertSign,cRLSign
CFG
cat > "${ROTATION_ROOT}/candidate/leaf.cnf" <<'CFG'
[req]
distinguished_name = dn
req_extensions = v3_req
prompt = no
[dn]
CN = stella-ops.local
[v3_req]
subjectAltName = DNS:stella-ops.local,DNS:*.stella-ops.local,IP:127.1.0.1,IP:127.0.0.1
[v3_leaf]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
subjectAltName = DNS:stella-ops.local,DNS:*.stella-ops.local,IP:127.1.0.1,IP:127.0.0.1
basicConstraints = critical,CA:FALSE
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth
CFG
openssl req -x509 -newkey rsa:3072 -nodes -days 3650 \
  -keyout "${ROTATION_ROOT}/candidate/kestrel-next-ca.key" \
  -out "${ROTATION_ROOT}/candidate/kestrel-next-ca.crt" \
  -config "${ROTATION_ROOT}/candidate/ca.cnf"
openssl req -new -newkey rsa:2048 -nodes \
  -keyout "${ROTATION_ROOT}/candidate/kestrel-next.key" \
  -out "${ROTATION_ROOT}/candidate/kestrel-next.csr" \
  -config "${ROTATION_ROOT}/candidate/leaf.cnf"
openssl x509 -req -in "${ROTATION_ROOT}/candidate/kestrel-next.csr" \
  -CA "${ROTATION_ROOT}/candidate/kestrel-next-ca.crt" \
  -CAkey "${ROTATION_ROOT}/candidate/kestrel-next-ca.key" -CAcreateserial \
  -days 825 -sha256 -extfile "${ROTATION_ROOT}/candidate/leaf.cnf" \
  -extensions v3_leaf -out "${ROTATION_ROOT}/candidate/kestrel-next.crt"
openssl pkcs12 -export \
  -out "${ROTATION_ROOT}/candidate/kestrel-next.pfx" \
  -inkey "${ROTATION_ROOT}/candidate/kestrel-next.key" \
  -in "${ROTATION_ROOT}/candidate/kestrel-next.crt" \
  -certfile "${ROTATION_ROOT}/candidate/kestrel-next-ca.crt" \
  -passout env:NEW_KESTREL_CERT_PASSWORD

This procedure treats kestrel-next-ca.key as a one-window ephemeral issuer and deletes it only after rollback retention expires. The helper does the same more aggressively: it has no retained-CA mode and destroys the issuer key at exit. Therefore a later helper invocation rotates the root and requires another dual-trust window. A future leaf-only reissue with no bundle change is possible only if Product/Security explicitly chooses a custody-retained lab CA key (for example, a separately approved Vault/operator secret) and provides an issuance procedure; that is not delivered here.

For a genuinely fresh local/dev install where kestrel-dev.pfx does not exist, ordinary helper generation remains non-destructive and needs no acknowledgement:

KESTREL_CERT_PASSWORD="${NEW_KESTREL_CERT_PASSWORD}" \
  devops/compose/scripts/ensure-dev-certs.sh

Non-development/operator-issued candidate

Obtain the certificate through the operator’s approved CA/PKI process. The delivered PFX must contain the matching private key and leaf certificate; stage the issuing intermediates and root as a separate ${ROTATION_ROOT}/candidate/kestrel-next-ca.crt trust bundle for the commands below. Require:

every external and internal DNS name actually used by clients in Subject Alternative Name;
serverAuth EKU, an approved key/algorithm/size, and the agreed validity period;
the exact issuer chain and revocation/status material required by operator policy; and
a password stored as a new secret-store version, never a tracked .env value.

Do not use the self-signed helper in a non-development deployment. Set the real mounted filename through KESTREL_CERT_PATH and, if Authority differs, KESTREL_AUTHORITY_CERT_PATH.

Candidate validation for both paths

openssl pkcs12 -in "${ROTATION_ROOT}/candidate/kestrel-next.pfx" \
  -passin env:NEW_KESTREL_CERT_PASSWORD -noout
openssl pkcs12 -in "${ROTATION_ROOT}/candidate/kestrel-next.pfx" \
  -passin env:NEW_KESTREL_CERT_PASSWORD -clcerts -nokeys \
  -out "${ROTATION_ROOT}/candidate/leaf.crt"
openssl x509 -in "${ROTATION_ROOT}/candidate/leaf.crt" \
  -noout -sha256 -fingerprint -serial -subject -issuer -dates -ext subjectAltName,extendedKeyUsage
openssl x509 -in "${ROTATION_ROOT}/candidate/leaf.crt" -purpose | grep 'SSL server : Yes'
openssl x509 -in "${ROTATION_ROOT}/candidate/leaf.crt" -checkend 2592000 -noout
openssl x509 -in "${ROTATION_ROOT}/candidate/kestrel-next-ca.crt" -noout -text \
  | grep 'CA:TRUE'
openssl x509 -in "${ROTATION_ROOT}/candidate/leaf.crt" -noout -text \
  | grep 'CA:FALSE'
openssl verify -purpose sslserver \
  -CAfile "${ROTATION_ROOT}/candidate/kestrel-next-ca.crt" \
  "${ROTATION_ROOT}/candidate/leaf.crt"

For an issued certificate, also run openssl verify -purpose sslserver -CAfile <candidate-ca-bundle> <candidate-leaf>. A missing intermediate is a candidate failure, not something to fix by disabling client validation.

OpenSSL structure checks are necessary but not final acceptance. Before phase 1, complete an actual TLS handshake from a pinned, already-local Linux image using the same .NET/runtime trust posture as a production consumer. Mount only the candidate CA and connect to an isolated server presenting the candidate leaf+chain. Record the Linux image ID, exit status, and peer fingerprint. Windows Schannel curl, Invoke-WebRequest, or a PowerShell pipeline is not equivalent evidence and cannot satisfy this gate. Do not pull a probe image during the window; verify it is already local and pinned.

3. Back up and write the rollback before cutover

Copy the exact mounted certificate files, both source trust bundles, and local/dev .env (if it is the secret source) into the protected backup directory. Preserve permissions and never add the directory to git.

for name in kestrel-dev.pfx kestrel-dev.key kestrel-dev.crt; do
  [ ! -e "${CERT_DIR}/${name}" ] || cp -p "${CERT_DIR}/${name}" "${ROTATION_ROOT}/backup/${name}"
done
cp -p "${COMBINED_CA_BUNDLE}" "${ROTATION_ROOT}/backup/combined-ca-bundle.crt"
cp -p "${GATEWAY_CA_BUNDLE}" "${ROTATION_ROOT}/backup/gateway-ca-bundle.crt"
cp -p "${AUTHORITY_CA_PIN}" "${ROTATION_ROOT}/backup/authority-ca.crt"
cp -p "${STAGED_COMBINED_CA_BUNDLE}" \
  "${ROTATION_ROOT}/backup/release-combined-ca-bundle.crt"
cp -p "${STAGED_GATEWAY_CA_BUNDLE}" \
  "${ROTATION_ROOT}/backup/release-gateway-ca-bundle.crt"
[ ! -f "${REPO_ROOT}/devops/compose/.env" ] || \
  cp -p "${REPO_ROOT}/devops/compose/.env" "${ROTATION_ROOT}/backup/compose.env"
[ ! -f "${REPO_ROOT}/devops/compose/.env.database-moves" ] || \
  cp -p "${REPO_ROOT}/devops/compose/.env.database-moves" \
    "${ROTATION_ROOT}/backup/compose.env.database-moves"

Record the current secret-store version ID and the exact command that makes it current again. Read it back before proceeding. A prose statement such as “restore from Vault” is not rollback.
Tag each target’s current image ID under a window-specific local rollback tag and verify the tag resolves to the same ID. These tags prevent certificate recreation from advancing unrelated application code through a moved :dev tag.

Put these exact file actions in the window rollback block (adjust filenames only if the deployment uses explicit KESTREL_*_CERT_PATH overrides):

install -m 0644 "${ROTATION_ROOT}/backup/kestrel-dev.pfx" "${CERT_DIR}/.kestrel-dev.pfx.rollback"
mv -f "${CERT_DIR}/.kestrel-dev.pfx.rollback" "${CERT_DIR}/kestrel-dev.pfx"
[ ! -f "${ROTATION_ROOT}/backup/kestrel-dev.key" ] || install -m 0644 \
  "${ROTATION_ROOT}/backup/kestrel-dev.key" "${CERT_DIR}/kestrel-dev.key"
[ ! -f "${ROTATION_ROOT}/backup/kestrel-dev.crt" ] || install -m 0644 \
  "${ROTATION_ROOT}/backup/kestrel-dev.crt" "${CERT_DIR}/kestrel-dev.crt"
install -m 0644 "${ROTATION_ROOT}/backup/combined-ca-bundle.crt" \
  "${COMBINED_CA_BUNDLE}"
install -m 0644 "${ROTATION_ROOT}/backup/gateway-ca-bundle.crt" \
  "${GATEWAY_CA_BUNDLE}"
install -m 0644 "${ROTATION_ROOT}/backup/authority-ca.crt" "${AUTHORITY_CA_PIN}"
install -m 0644 "${ROTATION_ROOT}/backup/release-combined-ca-bundle.crt" \
  "${STAGED_COMBINED_CA_BUNDLE}"
install -m 0644 "${ROTATION_ROOT}/backup/release-gateway-ca-bundle.crt" \
  "${STAGED_GATEWAY_CA_BUNDLE}"
[ ! -f "${ROTATION_ROOT}/backup/compose.env" ] || install -m 0600 \
  "${ROTATION_ROOT}/backup/compose.env" "${REPO_ROOT}/devops/compose/.env"
[ ! -f "${ROTATION_ROOT}/backup/compose.env.database-moves" ] || install -m 0600 \
  "${ROTATION_ROOT}/backup/compose.env.database-moves" \
    "${REPO_ROOT}/devops/compose/.env.database-moves"
# Run the recorded secret-store rollback command here, then read back its version.

0644 above matches the throwaway local/dev bind-mount requirement for UID 10001. A non-development deployment must use its operator-approved owner/group/mode or secret-volume ACL; never make a production private key world-readable to copy this example.

Before any phase, render each target with its recorded ordered env-file arguments and config-file chain, append an override pinning that one service to its current verified image ID, and read the protected render. Do not run a whole-chain up, use --remove-orphans, or hand-build a union. Every recreate names exactly one service and replays the same env/config arrays plus --no-deps --force-recreate. Delete rendered output after review because it contains secrets.

Use this dependency order wherever a phase says to recreate a consumer set:

Tier	Roles	Gate before next tier
1	Authority / identity issuer	Direct TLS and OIDC discovery succeed; a token can be issued.
2	Platform and foundational custody/configuration APIs	Each service is healthy and direct authenticated reads succeed.
3	Remaining internal owner APIs/web roles	Expected fingerprint/trust state is observed; no certificate-load or `UntrustedRoot` errors.
4	Background workers, schedulers, agents, and other outbound clients	Normal authentication and heartbeat/poll cycles succeed.
5	Router gateway/front door	External health, route discovery, login/token, and one normal gateway API read succeed.

Within a tier follow the rendered depends_on and runtime-owner relationships. Stop on the first failure rather than continuing to make the estate uniformly red.

4. Phase 1 — dual-trust expansion and trust-consumer recreate

The old PFX/password remain installed throughout this phase. Build two independent staged bundles, one from each original source bundle. Preserve every unrelated trust certificate byte-for-byte, normalize the managed block, and put both old and new trust material inside it:

local/dev legacy transition: retain the proven current self-signed CA:FALSE leaf temporarily, then add kestrel-next-ca.crt; this is the only phase where the invalid legacy leaf remains;
already-correct local/dev: retain the current CA:TRUE root, then add the next root; or
operator-issued: retain the approved old CA chain, then add the approved next CA chain. Never add an issued server leaf merely to make validation green.

A zero-marker bundle discovered in preflight is normalized here only after the current leaf and every old trust certificate have been fingerprinted. A partial/multiple marker still refuses. Do not use a broad subject/CN match: phase 1 must pull exact old/stale pins into the one managed block, and contraction must later remove those exact fingerprints, including every unmarked duplicate.

For each staged bundle require exactly one marker pair, prove that the exact old trust fingerprints remain, prove that every new trust fingerprint occurs once, and run:

openssl verify -purpose sslserver \
  -CAfile "${ROTATION_ROOT}/candidate/combined-ca-bundle.dual.crt" \
  "${ROTATION_ROOT}/candidate/leaf.crt"
openssl verify -purpose sslserver \
  -CAfile "${ROTATION_ROOT}/candidate/gateway-ca-bundle.dual.crt" \
  "${ROTATION_ROOT}/candidate/leaf.crt"

Then install both staged bundles through same-filesystem temporary names and atomic renames. Do not change the PFX or password yet. Recreate every row in trust-consumers.psvin dependency order, using its exact chain and current-image pin. Gate phase 2 on all of the following:

the currently served old fingerprint still works from every endpoint;
every recreated trust consumer mounts the expected dual-bundle digest;
a pinned Linux/.NET-equivalent consumer validates an isolated handshake to the candidate leaf;
the old PFX/password positive control still succeeds; and
no image ID, chain file, service-health baseline, OIDC flow, or normal authenticated read drifted.

authority-ca.crt is a third tracked pin but is not a managed-marker source. Prove its live mount count from the same container inventory. If any current consumer mounts it, stage it as old+new and include those consumers in both trust recreates. If its live mount count is zero, leave it unchanged until phase 3; do not manufacture a consumer from a stale compose fragment.

5. Phase 2 — PFX/password swap and PFX-consumer recreate

Treat the certificate-file and password switch as one bounded, estate-locked cutover transaction. No other recreate or deploy may interleave. Stage all candidate files on the destination filesystem, validate their digests, then rename each file into place (PFX last). Activate the new password secret/version and atomically replace its ignored runtime injection in the same transaction; do not recreate anything until the installed PFX opens with the read-back new value and fails with the old value. For local/dev, the file half is:

install -m 0644 "${ROTATION_ROOT}/candidate/kestrel-next.key" "${CERT_DIR}/.kestrel-dev.key.next"
install -m 0644 "${ROTATION_ROOT}/candidate/kestrel-next.crt" "${CERT_DIR}/.kestrel-dev.crt.next"
install -m 0644 "${ROTATION_ROOT}/candidate/kestrel-next.pfx" "${CERT_DIR}/.kestrel-dev.pfx.next"
mv -f "${CERT_DIR}/.kestrel-dev.key.next" "${CERT_DIR}/kestrel-dev.key"
mv -f "${CERT_DIR}/.kestrel-dev.crt.next" "${CERT_DIR}/kestrel-dev.crt"
mv -f "${CERT_DIR}/.kestrel-dev.pfx.next" "${CERT_DIR}/kestrel-dev.pfx"

Activate the new password secret version and update the ignored runtime injection. If mount or filename changes, update STELLAOPS_CERT_VOLUME, KESTREL_CERT_PATH, and KESTREL_AUTHORITY_CERT_PATH coherently. Read values back without printing them. If either the secret activation/injection or PFX validation fails, restore the old secret version and all backed-up certificate files before touching a container.

Recreate every current non-retired row in pfx-consumers.psvin dependency order with its exact ordered env-file list, config chain, and current-image pin. This includes HTTP-advertising roles and workers; it is not limited to socket listeners. In particular, SBOM must replay .env followed by .env.database-moves. The dual bundles remain installed throughout. Gate phase 3 on the new PFX/password positive, old-password negative, candidate/served fingerprint equality at every endpoint, expiry, OIDC/token flow, authenticated gateway read, and clean certificate/trust logs.

6. Phase 3 — old-trust contraction and trust-consumer recreate

Rebuild both bundles out of place from their phase-1 dual versions. Keep unrelated roots and only the approved next CA chain in the managed block. Remove every certificate whose exact fingerprint matches the legacy current leaf, legacy root, or stale duplicate recorded during preflight, even if it was outside the old marker. Do not remove by CN/subject and do not remove old trust before all phase-2 endpoints serve the new fingerprint.

For both contraction candidates prove:

exactly one marker pair and exactly one approved next trust chain;
none of the recorded old/stale fingerprints remains anywhere in the bundle;
the new leaf validates with openssl verify -purpose sslserver; and
a pinned Linux/.NET-equivalent handshake succeeds while a legacy self-signed leaf fails.

Atomically install both contracted bundles, then recreate every row in trust-consumers.psvagain in dependency order with exact image pins. Only after the full forcing-function set passes may the operator close the window and start the separately approved rollback-retention clock.

Replace the non-marker authority-ca.crt pin with the approved next CA:TRUE root through a same-filesystem temporary file and atomic rename (or contract its dual form if it was live-mounted). Then regenerate, never hand-edit, the two release-bundle copies and prove all five tracked pins:

install -m 0644 "${ROTATION_ROOT}/candidate/kestrel-next-ca.crt" \
  "${AUTHORITY_CA_PIN}.next"
mv -f "${AUTHORITY_CA_PIN}.next" "${AUTHORITY_CA_PIN}"
python devops/release/bundle/tools/stage-config.py
python devops/release/bundle/tools/stage-config.py --check
sha256sum "${PINNING_FILES[@]}" > "${ROTATION_ROOT}/inventory/final-pinning-files.sha256"

The served leaf fingerprint must be absent from every file in PINNING_FILES; the approved next root fingerprint must occur exactly once in each, and the two staged copies must be byte-identical to their sources. If authority-ca.crt had live consumers, recreate and gate them before closing.

7. Positive and negative controls

Run all applicable controls before closing the window:

New PFX/password positive: openssl pkcs12 -noout succeeds with NEW_KESTREL_CERT_PASSWORD.
Old password negative: the same command against the installed new PFX fails with OLD_KESTREL_CERT_PASSWORD. A success means the re-key criterion is not met.
Fingerprint equality: the candidate leaf SHA-256 fingerprint equals the leaf served by every public/internal TLS endpoint in the inventory. Obtain the served leaf with openssl s_client -connect <host:port> -servername <dns-name> </dev/null 2>/dev/null | openssl x509 -noout -sha256 -fingerprint -enddate.
Expiry: openssl x509 -checkend <operator-warning-seconds> -noout succeeds and the recorded notAfter matches the issued order. Thirty days is only a useful local warning, not a universal production policy.
Trust positive from both populations: a pinned Linux/.NET-equivalent client using the contracted combined-ca-bundle.crt, then gateway-ca-bundle.crt, completes a real TLS handshake without -k or an invalid-certificate bypass.
Old-trust negative: for a new local/dev root and issued leaf, the same request with the backed-up old bundle fails. This control is not expected to fail when old and new issued leaves share the same still-trusted CA; in that case prove that no endpoint serves the old fingerprint instead.
Application forcing function: OIDC discovery, a normal login or client-credentials token, and one authenticated gateway API read succeed. Container health alone is insufficient.
Bundle contraction: both bundle digests match the approved contracted candidates, new trust appears exactly once, and every recorded legacy/stale fingerprint is absent.
Estate comparison: trust-consumer and PFX-consumer counts equal preflight; every target matches its baseline disposition; and logs since recreate contain no certificate password, private-key, chain, UntrustedRoot, or metadata/JWKS retrieval failures.

7A. Five things measured in the first real execution (2026-08-22, PFX-3)

Point-in-time findings from the window that rotated the lab off the shared devpass passphrase (SPRINT_20260821_002 PFX-3). Each was verified by operation, and each cost time or an outage because the procedure above did not name it.

A third file pins the leaf, and two more are generated copies. Besides combined-ca-bundle.crt and gateway-ca-bundle.crt, the single-certificate devops/compose/authority-ca.crt held the outgoing leaf, and devops/release/bundle/config/{combined,gateway}-ca-bundle.crt are staged copies produced by python devops/release/bundle/tools/stage-config.py. Do not hand-edit the staged pair; regenerate it and confirm with --check. Find every pinning file by the outgoing leaf’s DER SHA-256, not by filename or marker.
kestrel-dev.crt / kestrel-dev.key were NOT the PFX’s identity. On disk they were an older, unrelated EC CN=*.stella-ops.localpair with no SAN and no extensions, while the live leaf inside kestrel-dev.pfx was RSA-2048 CN=stella-ops.local. The stale EC certificate was also what sat inside the # BEGIN/END stellaops-dev-kestrel-trust marker, and the live leaf sat unmarked elsewhere in the same file. Prove the three files are one identity before assuming it:
```
openssl x509 -in "${CERT_DIR}/kestrel-dev.crt" -pubkey -noout | sha256sum
openssl pkey  -in "${CERT_DIR}/kestrel-dev.key" -pubout       | sha256sum
```
ASPNETCORE_URLS does NOT tell you who loads the PFX. Only two services advertised an https:// URL, but 41 containers carried Kestrel__Certificates__Default__Password, and StellaOpsLocalHostnameExtensions.TryAddStellaOpsLocalBinding loads the PKCS#12 file for HTTP-advertising services too. A service left on the old password therefore crash-loops on X509CertificateLoader.LoadPkcs12FromFile even though it “only serves HTTP”. Scope the recreate by the password environment variable, never by the advertised URL scheme.
A per-service image-pin override CREATES a service that no longer exists. A retired service can leave a bare image:-only fragment in a still-referenced override file; adding a pin override for it makes compose materialise a brand-new default-named container on a brand-new default network, after destroying the orphan it replaced. Guard the pin by rendering the chain without the pin and refusing any service whose rendered definition has no container_name, or whose container_name is not the container you inventoried:
```
  docker compose -p <project> -f <chain…> config --format json \
    | jq -e --arg s <service> '.services[$s].container_name == ("stellaops-" + $s)'
```
doctor-web is the measured retired case. If an exited Doctor artifact is present, record it as an excluded retired/exited baseline; it is not one of the healthy targets and must not be recreated. The target-set equality check compares current non-retired inventories, not a historic hard-coded 27/41 count.
Mid-window UntrustedRoot is expected and must be timeboxed, not ignored. Between “clients recreated onto the new trust bundle” and “the gateway serves the new leaf”, clients calling the gateway log AuthenticationException … UntrustedRoot. In this window that produced 665 lines across three services and stopped 4 seconds before the gateway came up. The dual-trust expansion in section 4 exists to avoid this; if you skip it, assert the count returns to zero after the final recreate rather than treating the lines as noise.

8. Failure recovery

Failure before the phase-1 recreate: restore both backed-up bundles. The old PFX/password are still live, so do not restart the estate merely to abort.
Failure during phase 1: restore both original bundles and recreate only the already-touched trust consumers in dependency order with their exact pinned images.
Password/PFX mismatch or incomplete chain during recreate: stop immediately. Never set DangerousAcceptAnyServerCertificate, use curl -k as acceptance, or disable transport security.
Failure during phase 2: restore the old PFX/key/certificate and old secret version/injection, reinstall the phase-1 dual bundles, then recreate only the already-touched PFX consumers in dependency order with their exact env-file lists, config chains, and pinned images. Verify old fingerprint/password, health baseline, OIDC/token flow, and gateway read before reopening traffic. Retain dual trust until the retry or rollback closes.
Failure during phase 3: reinstall both phase-1 dual bundles, restore the phase-1 authority-ca.crt state and regenerate/check the staged copies, then recreate only already-touched trust consumers. The new PFX remains installed; do not roll it back solely because contraction failed.
Authority healthy but clients report UntrustedRoot: the mounted client bundle is stale or the client was not recreated. Compare its mount source and bundle digest to the installed bundle; fix forward only if the cause is proven, otherwise roll back.
A target image ID is missing: do not recreate it. Follow the live-deploy playbook’s export/import recovery, including restoring image metadata and plain-start verification, before rescheduling the certificate window.

Keep the old certificate, old secret version, all five trust/pinning files, image pins, and before/after evidence until the operator’s rollback-retention period expires. Then delete only the exact protected rotation directory and retire the old secret version according to custody policy. Retained evidence contains fingerprints, expiry, hashes, service/image IDs, and results — never private keys, passwords, tokens, or rendered compose secrets.

Finally, unset OLD_KESTREL_CERT_PASSWORD NEW_KESTREL_CERT_PASSWORD in the operator shell.