Backup and Restore Drill for One Installation

Audience: DevOps and QA operators rehearsing recovery of a Stella Ops installation. Owner: DevOps and QA roles. Status: procedure verified against source at 8ae42ab238 on 2026-10-10 (SPRINT_20261007_029 CCP-053) and revised the same day after review: the comparison baseline is recorded after the quiesce, and the commands cover a Windows host, the demo registry overlay and the installation’s own PostgreSQL image and user. It combines the supported cluster tools listed below. It has not been run end to end on a release-bundle installation. The first run is the CA-14 acceptance drill, which waits for an authorized disposable installation.

Retired on 2026-10-10: drill-backup-restore.sh and drill-backup-restore.ps1. The former pg_dump drill (DEVOPS-OPS-029-01) was removed. On a current cluster, where every service owns its own database (ADR-039), it could not do what it claimed:

  • It dumped one database (-d stellaops, or the database given with --db). It then removed the whole data volume, so every other service database was lost.
  • Its wipe step always stopped stellaops-postgres and removed compose_postgres-data, whichever container --container named. It then started postgres from the developer compose file.
  • Its round-trip check was one statement that reads authority.* and shared.tenants. No single database holds both schemas (stellaops_authority and stellaops_platform), so the statement printed nothing before and after. Two empty files compared equal, and the drill reported passed.

The drill below restores the whole cluster from an application-consistent backup. It restores the file-backed stores and the installation files with it, and it compares every table of every database.

Purpose

Prove that one named installation can be backed up, lost and restored, and that the restored installation serves the same data and verifies the same evidence. The drill covers:

When to run

Tools

The scripts are DevOps scripts in devops/compose/scripts/ of the repository. The release bundle does not ship them. Copy them to the operator host and record the repository revision you copied.

ScriptWhat it doesRefuses (exit code)
backup-postgres.shApplication-consistent backup of the whole cluster with pg_basebackup. Writes an artifact directory and a .stellaops-backup.json sidecar beside it that records the stop LSN.Source container missing or not running (2); the replication pre-flight fails (3)
verify-backup.shChecks the artifact against PostgreSQL’s own backup_manifest with pg_verifybackup.Missing manifest or member (3); checksum or manifest failure (4)
restore-postgres.shStages the artifact in a target volume. PostgreSQL replays the backup’s WAL on its next start.The developer estate’s data volumes, unless --i-am-restoring-the-live-estate is given; a volume that a running container mounts; a non-empty volume without --force (all 3)
pg-census.shWrites one sorted database|schema|table|rows line for every table in every database.Container not running (2)

The operations runbook describes each tool in detail: backup-restore-ops.md (SP-001A, SP-002, SP-003).

1. Name the installation

A release-bundle installation renders every Docker name from four values in its .env (bundle README.md, troubleshooting entry “The installer stops with ‘this installation would take Docker names that already belong to another installation’”). Work from the installation directory.

The commands are bash. On a Linux host, run them as written. On a Windows host, run them in Git Bash: the scripts in “Tools” handle Git Bash paths, and INSTALL_LABEL below converts the directory to the Windows form that Compose records. The drill has not run on either host yet.

TOOLS=/path/to/copied/devops/compose/scripts   # the scripts from "Tools"
cd /path/to/installation
INSTALL_DIR="$(pwd)"
# The directory as Compose records it in each container's working_dir label:
# a Windows path on a Windows host (Git Bash), the same path on Linux
INSTALL_LABEL="$(cygpath -w "$INSTALL_DIR" 2>/dev/null || printf '%s' "$INSTALL_DIR")"
env_value() { grep -E "^$1=" .env | tail -n 1 | cut -d= -f2- | tr -d '\r'; }
VOL_PREFIX="$(env_value STELLA_VOLUME_PREFIX)"; VOL_PREFIX="${VOL_PREFIX:-stellaops}"
ESTATE_PREFIX="$(env_value STELLAOPS_ESTATE_PREFIX)"; ESTATE_PREFIX="${ESTATE_PREFIX:-stellaops}"
PG_USER="$(env_value POSTGRES_USER)"; PG_USER="${PG_USER:-stellaops}"
PG_CONTAINER="${ESTATE_PREFIX}-postgres"
PG_VOLUME="${VOL_PREFIX}_postgres-data"
COMPOSE_FILES=(-f docker-compose.pinned.yml -f secrets/docker-compose.installation.yml)
# An installation made with --with-demo-registry (install.ps1: -WithDemoRegistry) adds the
# demo registry overlay to every compose call; its volume exists only then
if docker volume inspect "${VOL_PREFIX}_demo-registry-data" >/dev/null 2>&1; then
  COMPOSE_FILES+=(-f docker-compose.demo-registry.yml)
fi
compose() { docker compose --env-file .env "${COMPOSE_FILES[@]}" "$@"; }
export MSYS_NO_PATHCONV=1   # Git Bash only; it rewrites container paths otherwise

If the installation was started with these variables exported in the shell, a shell value overrode .env; use the values that were in effect. Without the demo registry overlay, compose would leave the demo-registry-data volume out of the inventory, the backup and the removal in section 5.

Confirm the pair before going further. Stop if either check prints something else:

# Must print the value of $PG_VOLUME
docker inspect "$PG_CONTAINER" \
  --format '{{range .Mounts}}{{if eq .Destination "/var/lib/postgresql/data"}}{{.Name}}{{end}}{{end}}'
# Must print the value of $INSTALL_LABEL
docker inspect "$PG_CONTAINER" \
  --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}'

2. Inventory what the installation holds

ClassWhereHow it is backed up
PostgreSQL clustervolume <prefix>_postgres-databackup-postgres.sh (section 4, step 3)
File-backed storesthe other named volumes of the installation, except caches and plugin scratchvolume archives (section 4, step 4)
Installation filesthe installation directorydirectory archive (section 4, step 5)

List the file-backed stores from the installation’s own compose model:

FILE_VOLUMES="$(compose config --volumes \
  | grep -v -E -- '-plugin-scratch$|cache(-data)?$|^postgres-data$')"
echo "$FILE_VOLUMES"

The filter leaves out volumes whose content the services rebuild: the scanner caches, the agent’s registry cache and the plugin scratch areas. Among the remaining volumes, these hold material that evidence verification depends on (source: docker-compose.pinned.yml of the bundle, devops/compose/docker-compose.evidence.yml, devops/docker/tsa/entrypoint.sh):

VolumeMounted byHolds
evidence-dataevidence-web, evidence-worker at /data/evidencethe Evidence object store (EvidenceLocker__ObjectStore__Kind=FileSystem)
attestor-proofchain-casevidence-web, evidence-worker at /app/data/proofchain-casthe proof-chain content-addressed store
tsa-datatsa at /var/lib/tsathe local timestamp authority’s root CA, signer key and serial counter, generated on first start. Losing them changes the timestamp trust root.
exportcenter-object-store-dataofflinekit-web, offlinekit-workerexported audit bundles

The installation directory holds .env (database role passwords, client secrets, KEKs), secrets/ (generated keys and certificates, the CA bundles, the agent CA keystore and its passphrase, the installation compose overlay), the generated keys under config/etc/ (listed in the bundle README.md, “Installer-generated secrets and keys”) and state/ (the installer’s records). The cluster stores every service role with the password that .env holds. Restore the directory and the cluster from the same backup set.

The directory archive contains every secret of the installation. Keep it under the same custody as the KEKs. stellaops-backup-crypto encrypt can wrap any archive (backup-restore-ops.md, SP-001 step 4).

3. Record the running installation

Run these while every service runs. Keep every output beside the backup set, never inside the postgres-basebackup-<ts> directory: pg_verifybackup fails on any file its manifest does not list.

BACKUP_DIR=/path/to/backup-set   # outside the installation directory
mkdir -p "$BACKUP_DIR/volumes" "$BACKUP_DIR/records"
PG_IMAGE="$(docker inspect "$PG_CONTAINER" --format '{{.Config.Image}}')"

# Identity: bundle manifest, and the image every service runs
sha256sum release-manifest.yaml > "$BACKUP_DIR/records/identity.txt"
compose ps --format '{{.Name}} {{.Image}}' >> "$BACKUP_DIR/records/identity.txt"
docker ps --filter "label=com.docker.compose.project.working_dir=$INSTALL_LABEL" \
  --format '{{.Names}}' | xargs -r docker inspect --format '{{.Name}} {{.Image}}' \
  >> "$BACKUP_DIR/records/identity.txt"

Record the evidence results. Retrieve and verify representative evidence through Evidence’s own interfaces, as in DR-003 of backup-restore-ops.md: the doctor checks, an exported bundle verified offline, and a capsule replay. Record every result, including those that already fail before the backup. Record one administrator sign-in.

These actions write rows and files: an exported bundle lands in exportcenter-object-store-data, each timestamp advances the serial counter in tsa-data, and a sign-in can write Authority rows. They therefore happen before the quiesce, and the comparison baseline is recorded after it (section 4, step 2).

4. Back up

  1. Quiesce. Stop every service except PostgreSQL, so the database rows and the file stores describe the same moment:

    compose stop $(compose config --services | grep -vx postgres)

    Stopping a service can still write: Valkey, for example, flushes its append-only file on stop. From here until the services start again in step 6, PostgreSQL alone runs: no service writes rows or file-store content, and PostgreSQL’s own data files are covered by the cluster backup, not by the checksum lists.

  2. Record the comparison baseline. The backup captures this quiesced state, so the restored installation must match it exactly (section 7):

    # Every table of every database
"$TOOLS"/pg-census.sh --container "$PG_CONTAINER" --user "$PG_USER" \
  > "$BACKUP_DIR/records/census-before.txt"

# Content of every file-backed store (the installation's own PostgreSQL image, no network)
for v in $FILE_VOLUMES; do
  docker run --rm --network none -v "${VOL_PREFIX}_${v}:/v:ro" "$PG_IMAGE" \
    sh -c 'cd /v && find . -type f -print0 | sort -z | xargs -0r sha256sum' \
    > "$BACKUP_DIR/records/sha256-before-${v}.txt"
done

    A census or checksum list taken while services run is not a valid baseline: every write between it and the quiesce is in the backup and shows up as a difference after the restore.

  3. Back up and verify the cluster:

    "$TOOLS"/backup-postgres.sh --container "$PG_CONTAINER" --user "$PG_USER" \
  --out-dir "$BACKUP_DIR" --yes
"$TOOLS"/verify-backup.sh --image "$PG_IMAGE" "$BACKUP_DIR"/postgres-basebackup-<ts>

  4. Archive each file-backed store:

    for v in $FILE_VOLUMES; do
  docker run --rm --network none -v "${VOL_PREFIX}_${v}:/v:ro" \
    -v "$BACKUP_DIR/volumes:/out" "$PG_IMAGE" tar -C /v -czf "/out/${v}.tar.gz" .
done

  5. Archive the installation directory:

    tar -C "$(dirname "$INSTALL_DIR")" -czf "$BACKUP_DIR/installation.tar.gz" \
  "$(basename "$INSTALL_DIR")"

  6. Record checksums of the backup set, then start the services again:

    (cd "$BACKUP_DIR" && sha256sum installation.tar.gz volumes/*.tar.gz \
  > records/backup-set.sha256)
compose start

5. Lose the installation (drill only)

compose config --volumes            # the volumes the next command removes
compose down -v
docker volume ls --filter "name=^${VOL_PREFIX}_" --format '{{.Name}}'   # must print nothing
cd "$(dirname "$INSTALL_DIR")"      # Windows refuses to move the current directory
mv "$INSTALL_DIR" "${INSTALL_DIR}.lost-$(date -u +%Y%m%dT%H%M%SZ)"

down -v removes the containers, networks and named volumes that this installation declares, and nothing else. Do not use docker volume prune or docker system prune: they act on every installation on the host.

6. Restore

  1. Restore the installation directory to the same absolute path. The checks in section 1 and the records in section 3 name this path, and each container that Compose creates records it. Another path also works for the installer once no container of the installation exists, as after section 5: it accepts the volumes that the directory’s state/installation-names records (install.sh, assert_installation_isolated; bundle README.md, the same troubleshooting entry as in section 1, on moving an installation). A different path makes the records of this drill harder to compare, so keep the original one.

    tar -C "$(dirname "$INSTALL_DIR")" -xzf "$BACKUP_DIR/installation.tar.gz"
cd "$INSTALL_DIR"

  2. Let Compose create the containers and empty volumes without starting anything. A volume created with docker volume create has no Compose labels. The installer later refuses such a volume as “not created by docker compose”.

    compose up --no-start
PG_IMAGE="$(docker inspect "$PG_CONTAINER" --format '{{.Config.Image}}')"

  3. Stage the cluster. The volume is empty and no container runs, so the script needs neither --force nor an override. If it asks for one, stop and check the names again.

    "$TOOLS"/restore-postgres.sh --image "$PG_IMAGE" \
  --artifact "$BACKUP_DIR"/postgres-basebackup-<ts> --target-volume "$PG_VOLUME" --yes

    --image keeps the restore on the installation’s digest-pinned PostgreSQL image. Without it, and without jq on the host, the script falls back to postgres:18.1, which an offline host may not hold.

  4. Extract each file-backed store into its new volume:

    for v in $FILE_VOLUMES; do
  docker run --rm --network none -v "${VOL_PREFIX}_${v}:/v" \
    -v "$BACKUP_DIR/volumes:/in:ro" "$PG_IMAGE" tar -C /v -xzf "/in/${v}.tar.gz"
done

  5. Start PostgreSQL alone and read its recovery:

    compose up -d postgres
docker logs "$PG_CONTAINER" 2>&1 \
  | grep -E 'backup recovery|consistent recovery state|ready to accept'

    The LSN in consistent recovery state reached at <LSN> must equal stopLsn in the artifact’s .stellaops-backup.json. The log says backup recovery, not crash recovery.

  6. Take the after-state records before any other service starts, because running services write rows:

    "$TOOLS"/pg-census.sh --container "$PG_CONTAINER" --user "$PG_USER" \
  > "$BACKUP_DIR/records/census-after.txt"
for v in $FILE_VOLUMES; do
  docker run --rm --network none -v "${VOL_PREFIX}_${v}:/v:ro" "$PG_IMAGE" \
    sh -c 'cd /v && find . -type f -print0 | sort -z | xargs -0r sha256sum' \
    > "$BACKUP_DIR/records/sha256-after-${v}.txt"
done

  7. Start the installation: compose up -d, then wait until compose ps reports every service healthy.

7. Verify

8. Record the drill

ItemRecord
Installationbundle commit and release-manifest.yaml sha256; installation prefixes; the image of every service (records/identity.txt)
HostCPU, memory, free disk before and after; Docker and Compose versions; scripts’ repository revision
SizesPostgreSQL data size; the size of each file-backed volume; the size of each artifact
Timewall-clock time of each phase: quiesce, cluster backup, volume and directory archives, loss, each restore step, recovery, all services healthy, first administrator sign-in
Operator effortevery command run and every manual decision or correction, counted
Outcomesrecovery LSN against stopLsn; census diff; file-store diffs; evidence results before and after; each failure with its exact message

Measured times describe this installation and its data volume only. Do not quote them as the recovery time of another installation. See “Recovery objectives (measured)” in backup-restore-ops.md.

What the drill does not prove

Current coverage and remaining limits (2026-10-20)

Archive source qualified at d8fc7e65d7; re-verify with python devops/compose/scripts/test_backup_evidence.py -v. This does not qualify the complete installation drill described above.

Developer compose estate

The developer estate on a workstation uses fixed physical names (stellaops-postgres, compose_postgres-data, compose_evidence-data and others; devops/compose/AGENTS.md). restore-postgres.sh refuses its data volumes unless --i-am-restoring-the-live-estate is given. Do not run this drill on that estate. Restore it only as a recovery, in an approved window.

Cross-references