Runbook: Backup and Restore Operations

Sprint: SPRINT_20260117_029_Runbook_coverage_expansion Task: RUN-004 - Backup/Restore Runbook

Purpose: Capture, verify, and restore the Stella Ops compose stack — the PostgreSQL state volume, the RustFS object store, and Valkey — using the shipped DevOps scripts and product-native tooling. Audience: Platform / DevOps on-call performing backups, restore drills, or disaster recovery. Status: documents the shipped backup paths — an application-consistent PostgreSQL backup (backup-postgres.sh, since 2026-08-16) and the older crash-consistent volume snapshot (backup.sh), plus their restore and verification procedures; reconciled against source 2026-05-30, extended 2026-08-16 UTC. Sections marked NOT IMPLEMENTED below are roadmap, not shipped; plan recovery objectives against what this page verifies, not against the roadmap items. Measured RTO/RPO figures are in Recovery objectives — read the shape they were measured on before quoting them.

Reconciliation note (2026-05-30, re-verified 2026-05-31). This runbook was originally written against a planned stella backup … CLI surface (create / verify / restore / schedule / retention / PITR / offline). That CLI does not exist — there is no backup command group in src/Cli/StellaOps.Cli/Commands/CommandFactory.cs, no stella service lifecycle command group, and no stella db migrate subcommand (the stella db group that does exist only triggers Concelier connector jobs — fetch / merge / export). The real, shipped backup/restore mechanism is a set of DevOps shell scripts plus the product-native stellaops-backup-cryptoencryptor and the stella doctorstorage checks. Evidence-store operations are an exception: a real stella evidence command group does exist (export, verify, verify-offline, provenance, proof, store, status, card, …) — see DR-003. This document has been rewritten to match the implemented tooling. Aspirational items that are not yet implemented are explicitly marked NOT IMPLEMENTED or Draft/roadmap.

Scope

Backup and restore procedures for the Stella Ops compose stack: the PostgreSQL state volume, the RustFS object store, and Valkey. Covers volume-snapshot backups, optional product-native encryption, the disaster-recovery restore drill, and the doctor checks that watch backup health.

Canonical companion docs (read alongside this runbook):

• docs/runbooks/deploy/backup-restore-drill.md — the canonical pg_dump → wipe → restore drill (DEVOPS-OPS-029-01).
• docs/modules/evidence-locker/encrypted-backup-restore.md — the encrypted backup + independent key recovery design (Sprint 20260520_088 / TOPO-115).
• docs/doctor/articles/storage/backup-directory.md — the check.storage.backup doctor check article.
• docs/runbooks/database/migration-recovery.md and docs/architecture/decisions/ADR-004-forward-only-migrations.md — the forward-only migration recovery path that postgres restore relies on.

Backup Architecture Overview

What gets backed up

The shipped backup script (devops/compose/scripts/backup.sh) snapshots three live data volumes, addressed by their physical names:

Override PG_VOLUME / RUSTFS_VOLUME / VALKEY_VOLUME to back up a different stack’s volumes.

Changed 2026-08-15 — COMPOSE_PROJECT_NAME is no longer honoured here, on purpose. Every short-form volume in devops/compose/** now carries an explicit top-level name: (SPRINT_20260810_002 PTC-7 prerequisite), so volume identity is independent of the compose project name. The script previously derived ${COMPOSE_PROJECT_NAME:-compose}_postgres-data; under a project rename that derivation would have silently followed the rename onto brand-new EMPTY volumes and reported a successful backup of nothing. A hardcoded name fails loudly (exit 3, “required source volume(s) not found”); a correctly-parameterised one failed quietly.

Backup type. backup.sh produces a single crash-consistent tar.gz snapshot of the volumes. There is no continuous WAL archiving, no incremental/differential mode, and no separate per-component schedule shipped in the repo. PostgreSQL point-in-time recovery (PITR) is NOT IMPLEMENTED (see SP-004 below).

Since 2026-08-16 this is no longer the only path for PostgreSQL. backup-postgres.sh takes an application-consistent backup of the database cluster (SP-001A). Use it for PostgreSQL and keep backup.sh for RustFS and Valkey. The two consistency models are compared in the next section — the difference is not academic, and the buyer objection this runbook answers was specifically about it.

Two backup paths, and which consistency model each gives you

What “crash-consistent” actually means for the older path. It restores the way a machine restores after losing power: usually fine, never guaranteed, and never provable ahead of time. That is an acceptable model for RustFS blobs (immutable content-addressed objects) and for Valkey (cache and transport state that is rebuilt anyway). It is a weak model for a transactional database, which is why PostgreSQL now has its own path.

What the application-consistent path guarantees:

• On restore the cluster recovers to the stop LSN — the instant the copy finished. Every transaction committed at or before it is present; none after it.
• Every logical database recovers at that same point. ADR-039 puts all service databases on the shared installation server, and this is one physical copy of that server. A per-database pg_dump sweep does not give you this: each dump takes its own snapshot, so cross-service state can restore skewed.

What it does not guarantee:

• No point-in-time recovery. Recovering to an arbitrary moment needs continuous WAL archiving; archive_mode is off on this estate (verified 2026-08-16). The recovery point is the stop LSN of whichever backup you restore, so RPO is bounded by your backup interval, not by seconds (SP-004).
• It is not a correctness check on your data. A backup of a corrupted database restores that corruption faithfully.
• It does not cover RustFS or Valkey — keep taking those with backup.sh.

Worker pause and encryption (both OFF by default)

• Worker pause is optional and off by default (PAUSE_WORKERS=1 to opt in). PostgreSQL/RustFS/Valkey ship crash-consistent backups for an offline-tar snapshot; pausing live workers on a shared stack disrupts other operators.
• Encryption is optional and off by default (STELLAOPS_BACKUP_ENCRYPT=1 to opt in). When enabled, the plaintext tar.gz is run through the product-native stellaops-backup-crypto encryptor (AES-256-GCM envelope, opaque STLEBK01 artifact + HMAC-SHA256-signed manifest) and the plaintext is removed.

Storage locations

• Primary: the script writes to ./backups/ by default (override with the BACKUP_DIR environment variable, which may be an absolute path such as /var/lib/stellaops/backups/).
• Offline / air-gap: point BACKUP_DIR at removable media; enable encryption so the artifact at rest is opaque.
• Secondary cloud object storage (S3 / Azure Blob / GCS): NOT IMPLEMENTED and out of posture. Stella Ops is self-hosted / on-prem-first; there is no cloud-managed backup target wired in. Copy the local artifact to your own off-host storage with standard OS tooling.

Pre-flight Checklist

Environment verification

# Confirm the source volumes exist (dry-run prints the exact docker run it would issue)
./devops/compose/scripts/backup.sh --dry-run

# Verify backup directory accessibility + recent-backup presence (doctor check)
stella doctor run --check check.storage.backup

# Verify there is enough free disk for the snapshot
stella doctor run --check check.storage.diskspace

# Verify the database is reachable before a restore drill
stella doctor run --check check.postgres.connectivity

The check.storage.backup check reads its path from the Backup:Path or Storage:BackupPath configuration key, verifies the directory exists and is writable, scans for backup files (.bak, .backup, .tar, .tar.gz, .tgz, .zip, .sql, .dump), and warns if no backup is present or the most recent one is older than 7 days. It only runs when a backup path is configured. Source: src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Storage/Checks/BackupDirectoryCheck.cs.

What to watch

Backup health is observed in two places, and they answer different questions.

Pull — the Doctor check. The forcing function for backup freshness is doctor.platform.backup.schedule, served by platform-web’s /doctor/platform-web/checks and surfaced in the console through Platform’s doctor aggregate. It fails when a backup that should have happened did not. It answers when something asks.

Push — Notify alerts. As of BRP-4 the same facts are announced on the product’s own notification pipeline, so an operator learns about a backup gap without opening anything. See Backup alerts for the four event kinds, how to subscribe, and what to do on receipt. This is the half a pull-only check cannot cover: a backup that never happened generates no event by itself, so Platform polls the declared schedule and emits platform.backup.run-missed when the expectation is violated.

Prometheus metrics remain NOT IMPLEMENTED, and so does the Prometheus mirror of these alerts. No process emits stella_backup_last_success_timestamp, stella_backup_duration_seconds, stella_backup_size_bytes or stella_restore_test_last_success, and the alert rule names StellaBackupFailed / StellaBackupStorageFull still exist only in documentation. Keeping them named here is deliberate: they are the intended Prometheus mirror of the Notify rules below, for operators who integrate with their own alerting stack, and the rules cannot be written until there is a metric substrate and something that loads them. Verified 2026-08-16 and unchanged 2026-08-18: the telemetry overlay is not in the live estate’s config_files, it defines no alertmanager, and devops/telemetry/storage/prometheus.yaml has no rule_files stanza — so an alert rule written today could never fire. That is a declared non-goal for this sprint, not an oversight; it needs the observability stack deployed.

Corrected 2026-08-18 (BRP-3). This section previously named stella doctor run --check check.storage.backup as the freshness forcing function. That check cannot serve as one: it warns on a hardcoded seven-day threshold no policy can change, and since doctor-web was retired (2026-08-17, DOC-5 stage 4) the stellaops.doctor.storage bundle it lives in has no runtime loader — the CLI registers only the Core, Database and BinaryAnalysis plugins in-process. It is still built, signed and packaged; it is not run. Verified against src/ rather than assumed.

Events are emitted for BRP-4 to consume: every reported cycle publishes platform.backup.run-completed or platform.backup.run-failed onto the Notify event stream (best-effort — the ledger row is the record of truth, and a Notify outage never turns a good backup into a failed report). Choosing the delivery mechanism — Notify channels, a metrics path, or both — is BRP-4’s opening decision.

Standard Procedures

SP-001: Create a manual backup

When: Before upgrades, schema changes, or major configuration changes. Duration: Minutes, depending on volume size.

1. Preview the actions without touching anything:

   ./devops/compose/scripts/backup.sh --dry-run
   

2. Create a plaintext snapshot (interactive prompt; add --yes to skip):

   ./devops/compose/scripts/backup.sh --yes
   # writes ./backups/stellaops-backup-<UTC-timestamp>.tar.gz
   

3. Optional — quiesce the scanner worker during the tar (approved window only):

   PAUSE_WORKERS=1 ./devops/compose/scripts/backup.sh --yes
   

4. Optional — write an encrypted (opaque) artifact + signed manifest:

   STELLAOPS_BACKUP_ENCRYPT=1 \
   STELLAOPS_BACKUP_ENCRYPTION_KEY_FILE=/secure/backup-kek.key \
   STELLAOPS_BACKUP_SIGNING_KEY_FILE=/secure/backup-sign.key \
   ./devops/compose/scripts/backup.sh --yes
   # writes <archive>.tar.gz.stlebk + <archive>.tar.gz.stlebk.manifest.json
   # and removes the plaintext tar.gz
   

5. Custom output directory (e.g. removable media):

   BACKUP_DIR=/media/usb/stellaops ./devops/compose/scripts/backup.sh --yes
   

The KEK and the manifest-signing key are separate keys, recoverable independently of the running stack (file or env). FileBackupKeyProvider (key files) is preferred over the env path for air-gap key escrow. See docs/modules/evidence-locker/encrypted-backup-restore.md.

SP-001A: Create an application-consistent PostgreSQL backup

When: As the primary PostgreSQL backup — before upgrades and schema changes, and on whatever cadence your RPO requires (see Recovery objectives). Duration: Roughly 40 minutes for the current live cluster; see the measured rate below.

# Preview: pre-flight only (opens a replication connection, prints the plan, copies nothing)
./devops/compose/scripts/backup-postgres.sh --dry-run

# Take the backup
./devops/compose/scripts/backup-postgres.sh --yes

# Somewhere with room, e.g. off-host media
BACKUP_DIR=/mnt/backups ./devops/compose/scripts/backup-postgres.sh --yes

It writes an artifact directory plus two siblings:

<BACKUP_DIR>/postgres-basebackup-<ts>/          <- base.tar.gz, pg_wal.tar.gz, backup_manifest
<BACKUP_DIR>/postgres-basebackup-<ts>.stellaops-backup.json   <- recovery point, sizes, SHA256s
<BACKUP_DIR>/postgres-basebackup-<ts>.pg_basebackup.log

Never put anything inside the artifact directory. pg_verifybackup fails any tar-format backup containing a member its manifest does not list — “file X is not expected in a tar format backup”. Drop a note, a checksum file or a restore record in there and every future verification of an otherwise perfect backup fails. That is why the sidecar and log are siblings. verify-backup.sh reports this case as pollution rather than corruption, because the operator response is different.

How it reaches the server without changing anything. pg_basebackup needs a replication connection, and this estate’s pg_hba.conf allows those only from 127.0.0.1 — the catch-all host all all all scram-sha-256 line does not cover replication connections, because all in the database column deliberately excludes them. Rather than edit pg_hba.conf and restart the live server, the script runs pg_basebackup in a throwaway container that joins the postgres container’s network namespace (--network container:<source>), so its 127.0.0.1 is the server’s loopback. Nothing about the running estate changes.

Load it imposes (it is a read, but not a free one):

• A full sequential read of the data directory plus a second connection streaming WAL, for the whole duration. Measured at ~37 MB/s of source on the live estate (2026-08-16). Run it in a quiet window if ingest latency matters.
• By default it requests an immediate checkpoint, a bounded write burst at the start. --spread-checkpoint is gentler but the copy does not begin until the checkpoint completes — up to several minutes of apparent hang.
• It takes no locks on user tables and blocks no writes.

Prerequisites (all satisfied on this estate; the script pre-flights them and fails with the specific remedy before copying a single byte): wal_level >= replica, max_wal_senders > 0, a backup role with REPLICATION (or superuser), and a replication line in pg_hba.conf reachable from loopback.

NOT IMPLEMENTED for this path: the encryption envelope. backup.sh can emit an opaque STLEBK01 artifact + signed manifest (SP-001 step 4); backup-postgres.sh cannot yet. Until it can, protect the artifact directory with storage-level controls, or tar-and-encrypt it out of band.

SP-002: Verify a backup

Frequency: Weekly (and before any destructive restore). Duration: Minutes.

Use verify-backup.sh — it handles both artifact shapes and fails closed:

# Application-consistent artifact: the real check
./devops/compose/scripts/verify-backup.sh backups/postgres-basebackup-<ts>

# Crash-consistent volume tar: integrity only, and it says so
./devops/compose/scripts/verify-backup.sh backups/stellaops-backup-<ts>.tar.gz

Exit codes: 0 verified, 2 artifact not found or unrecognised, 3 structural failure (missing manifest or WAL member), 4 checksum/manifest verification failed — do not rely on this backup.

For a pg_basebackup artifact it re-checks the SHA256s recorded when the backup was taken, then runs pg_verifybackupagainst postgres’s own backup_manifest, which verifies every file’s size and SHA256 and validates the manifest’s own checksum first (so a tampered manifest cannot vouch for a tampered backup).

What a pass proves, and what it does not. It proves the artifact is intact and complete: bit rot, truncation, a half-finished copy and a silently-failed transfer all fail it. It does not prove the data is semantically correct — a backup of a corrupted database verifies perfectly, because it is a faithful copy of corruption. Only the periodic restore drill answers that.

Red-proved 2026-08-16 (a verification that cannot fail is worthless), all four cases rejected: a truncated base.tar.gz; a single flipped byte with the sidecar rewritten to match — caught by pg_verifybackup naming the damaged relation, which proves the check does not depend on our own metadata; a removed pg_wal.tar.gz; and a stray file inside the artifact directory.

Plaintext volume snapshots are plain tar.gz and can also be checked by hand:

gzip -t backups/stellaops-backup-<ts>.tar.gz   # integrity of the gzip stream
tar tzf backups/stellaops-backup-<ts>.tar.gz | head   # lists data/postgres, data/rustfs, data/valkey

Encrypted artifacts are verified by attempting a decrypt — the manifest HMAC signature and the ciphertext SHA-256 are checked before any plaintext is produced, and the tool fails closed on tamper / wrong key (exit code 3):

stellaops-backup-crypto decrypt \
  --in  backups/stellaops-backup-<ts>.tar.gz.stlebk \
  --out /tmp/verify.tar.gz \
  --manifest backups/stellaops-backup-<ts>.tar.gz.stlebk.manifest.json \
  --kek-file /secure/backup-kek.key \
  --signing-key-file /secure/backup-sign.key

The strongest verification remains the end-to-end restore drill (SP-003 / the DEVOPS-OPS-029-01 drill), which restores into a clean cluster and asserts a row-count round-trip — but it is no longer the only option, which was the point: a drill costs a reset window, so it does not happen often enough for “our backups are good” to mean anything between drills. verify-backup.sh is the between-drills answer.

Naming, not a gap (was previously flagged NOT IMPLEMENTED). There is still no stella backup verify CLI subcommand, nor --all-unverified / stella backup log-verification. The capability ships as devops/compose/scripts/verify-backup.sh (SPRINT_20260802_003 BRP-5), consistent with how the rest of this runbook’s backup tooling ships — as DevOps scripts rather than a stella backup command group. If the CLI surface is wanted later, it wraps this script; the verification itself is done.

SP-003: Restore from a backup

CAUTION: this is a destructive operation. Run only in an approved reset window.

The canonical, scripted restore path is the backup + restore drill, which pg_dumps the live cluster, wipes the volume, brings postgres back, and restores — asserting that representative row counts round-trip. See docs/runbooks/deploy/backup-restore-drill.md for the full operator checklist.

# Dry-run first (default mode; prints actions, touches nothing, exits 10)
./devops/compose/scripts/drill-backup-restore.sh

# Confirmed destructive run (approved reset window only)
./devops/compose/scripts/drill-backup-restore.sh --confirm

# Windows / PowerShell
.\devops\compose\scripts\drill-backup-restore.ps1 -Confirm

To restore an application-consistent artifact produced by backup-postgres.sh (the preferred PostgreSQL path):

# 1. Verify BEFORE destroying anything. If this fails you still have a running estate.
./devops/compose/scripts/verify-backup.sh backups/postgres-basebackup-<ts>

# 2. Stop the consumers of the target volume, then stage the cluster into it.
#    The script REFUSES a volume that a running container has mounted, refuses the
#    estate's live volumes without --i-am-restoring-the-live-estate, and refuses a
#    non-empty target without --force.
./devops/compose/scripts/restore-postgres.sh \
  --artifact backups/postgres-basebackup-<ts> \
  --target-volume compose_postgres-data \
  --i-am-restoring-the-live-estate --force --yes

# 3. Start postgres. Recovery happens HERE, not in step 2.
docker compose -f devops/compose/docker-compose.stella-ops.yml up -d postgres
docker logs stellaops-postgres 2>&1 | grep -E 'backup recovery|consistent recovery state|ready to accept'

Step 3’s log is the proof the restore worked, and it is worth reading rather than skipping. Expect these lines, in this order:

starting backup recovery with redo LSN <start>, checkpoint LSN <…>, on timeline ID 1
completed backup recovery with redo LSN <start> and end LSN <stop>
consistent recovery state reached at <stop>
database system is ready to accept connections

The <stop> LSN must equal the stopLsn in the artifact’s .stellaops-backup.json. Note it says backup recovery, not crash recovery — postgres found a backup_label and replayed the backup’s own WAL. A volume-tar restore cannot produce that line, and that difference is the whole reason this path exists.

Then continue with steps 4 and 5 below (service restart for HELLO replay, and the doctor verification).

To restore a volume snapshot produced by backup.sh (still the path for RustFS and Valkey, and the fallback for postgres):

1. Stop the stack:

   docker compose -f devops/compose/docker-compose.stella-ops.yml down
   

2. Wipe the existing postgres volume (canonical fresh-DB step — stops the container before docker volume rm so the removal cannot be soft-skipped):

   ./devops/compose/scripts/wipe-postgres.sh --yes
   

3. Restore the volume contents from the snapshot (decrypt first if encrypted), then bring the stack back up. Services auto-migrate their own schemas on startup (CLAUDE.md §2.7), so no manual migration step is required:

   docker compose -f devops/compose/docker-compose.stella-ops.yml up -d
   

4. After a gateway restart, restart all services so they re-send their HELLO via Valkey (transport metadata replay):

   docker compose -f devops/compose/docker-compose.stella-ops.yml restart
   

5. Verify restoration:

   stella doctor run --check check.postgres.connectivity
   stella doctor run --check check.postgres.migrations
   # then confirm admin login through https://stella-ops.local
   

NOT IMPLEMENTED: stella service stop/start/restart/health and stella db migrate. Use docker compose for lifecycle and the auto-migration-on-startup invariant instead of a manual migrate step. Amended 2026-08-16: stella backup restore still does not exist as a CLI subcommand, but the restore capability is no longer manual — it ships as devops/compose/scripts/restore-postgres.sh (SPRINT_20260802_003), with the structural safety guards described above. (Note: stella evidence verify does exist — it verifies an exported evidence bundle’s DSSE/Rekor signatures, not the postgres restore; see DR-003.)

SP-004: Point-in-Time Recovery (PITR)

Status: NOT IMPLEMENTED (roadmap).

There is no WAL archiving configured and no stella backup restore-pitr / wal-list command. Recovery granularity is the most recent volume snapshot (SP-001) or the most recent pg_dump captured by the drill. For point-in-time recovery you would need to enable PostgreSQL WAL archiving out of band; that is not part of the shipped tooling. Track any requirement for PITR as a new sprint task rather than relying on this section.

Recovery objectives (measured)

These are measured numbers from an executed exercise, not estimates. Read the shape they were measured on before quoting them; anything below labelled a projection is arithmetic on a measured rate, not a measured result.

The exercise — 2026-08-16 UTC

Full loss and recovery of a scratch installation: application-consistent PostgreSQL backup taken, all three data volumes destroyed, everything restored, round-trip verified. Evidence: docs/implplan/_evidence/2026-08-16-sprint-20260802-003-brp1-restore-exercise.json.

Installation shape it was measured on. PostgreSQL 18.1, 6 databases, 273 MiB of PGDATA, 112 user tables, 160,413 rows; RustFS 35 MB / 113 objects; Valkey 161 MB / ~2,100 keys. The databases were real data read out of the live cluster with pg_dump, so schemas, indexes and row distributions are production-shaped — only the volume is smaller. The live estate is 86.3 GiB, so do not quote the exercise RTO as the estate’s RTO; use the rate below.

Both RTO numbers are reported because they measure different things and neither is padded: 65 s is the sum of the mechanical phases; 133 s is what the operator experienced, including the gaps between three separately issued commands.

Round-trip verification after restore — all PASS:

• PostgreSQL: all 112 tables and 160,413 rows identical, whole-cluster census diff empty (pg-census.sh, which compares every user table in every database — the older drill compares four tables, and a restore that dropped every other schema would pass that).
• RustFS: 113 objects, every SHA256 unchanged.
• Valkey: the planted marker key survived. Key count moved 2,103 → 2,092, which is not restore loss — 1,366 keys carry TTLs and expired across the window. Valkey holds cache and transport state, so an exact key-count round-trip is not a valid restore assertion for it; a marker key is.
• Recovery point: the cluster reached consistent recovery state at 0/14000158 — byte-identical to the stop LSN recorded before the volumes were destroyed. That is a direct proof of the consistency model, not an inference from it.

That exercise proved the data came back. It could not prove the estate came back usable, because it restored storage only — postgres, RustFS and Valkey, with no service running against them. The application-layer half is the exercise below.

The application-layer exercise — 2026-08-17 UTC

Same loss and recovery, but the scratch installation included the services and the data included real evidence capsules, so the question it answers is not “did the bytes come back” but “does the estate still do its job”. Evidence: docs/implplan/_evidence/2026-08-17-sprint-20260802-003-brp1-capsule-replay.json.

Installation shape. PostgreSQL 18.1, 3 databases (postgres, stellaops_authority, stellaops_platform), 277 MiB of PGDATA, 97 user tables, 155,276 rows, plus the evidence object store. Running against it: authority and evidence-locker-web, both on the images the live estate runs and both configured by cloning the live containers’ configuration, so they ran the production shape (LocalHarness=false, signing on, timestamping required, Router on) rather than a relaxed harness. The data was real: four sealed decision capsules and the evidence bundles, exported read-only from the live cluster.

Read the 55 s carefully: 48 s of it was restore-postgres.sh’s running-consumer guard, not the restore. On this host the guard inspected all 51 running containers one at a time; the extraction itself took 2 s. That was fixed in the same change (the guard now asks the engine once, docker ps --filter volume=…: 0.4 s against 31.8 s, same answer, and still red-proved to refuse a volume with a live consumer). Expect the guard-inflated figure on any pre-fix copy of the script.

The capsule criterion — what was actually proven. The four capsules were replayed and DSSE-verified through evidence-locker-web twice: once before the backup, once after the total-loss restore. The replay responses are byte-identical across the two runs (SHA256 1c1e141c…), and the surviving sealed capsules verified with signatureValid: true against key id evidence-locker-capsule-key (ED25519) after the restore.

Two details keep that from being a self-congratulating check:

• The endpoint recomputes. POST /api/v1/evidence/capsules/{id}/replay re-canonicalises the stored manifest and recomputes the content hash. Flipping a single character inside a restored capsule made it return contentHashMatches:false immediately; flipping it back returned true. A 200 from this endpoint is an integrity assertion, not a row read.
• The dataset contains capsules that legitimately fail. Two of the four are hand-seeded demo rows carrying a placeholder DSSE signature, and they replay false — before the backup and after the restore alike. So the honest expectation after a restore is not “everything green”, it is “identical to the baseline, including the reds” — which is what a restore that silently regenerated or dropped rows could not produce.

Replaying a capsule needs a real token; plan for it. The capsule routes are tenant-scoped, and the internal bypass network is Strict: it substitutes for a missing identity but grants no tenant, and X-StellaOps-TenantId is honoured only for a global-admin principal. A tokenless call is therefore refused with 400 tenant_missing. A recovery drill that intends to verify evidence must restore Authority first and mint a token — and note that OpenIddict rejects every plain-HTTP token request with This server only accepts HTTPS requests (ID2083) regardless of RequireHttpsMetadata, so call the HTTPS listener.

Live-estate scale anchor — measured 2026-08-16 UTC

Rather than estimate the 86.3 GiB estate from a 273 MiB exercise, a bounded application-consistent read was run against the live cluster and then stopped:

The 40-minute and 17 GiB figures are projections from that measured rate, not a measured full run. A full production backup was deliberately not taken in that window: the vulnerabilities-worker was mid-ingest and a 40-minute sustained read plus ~17 GiB of writes was not worth imposing for a number the sample already characterises. The read was verified harmless — postgres stayed healthy, the worker was not interrupted, and no replication slot was left behind.

Projected live RTO for PostgreSQL is dominated by extract throughput, measured at roughly 5 MiB/s of artifact through the decompress-and-write path (30 MB in 58 s). Against a ~17 GiB artifact that is on the order of an hour, plus seconds of WAL recovery. This is an extrapolation and has not been measured at live scale — treat it as a planning figure and measure it in your own reset window.

RPO — state this one honestly

RPO equals the age of the newest verified backup at the moment of loss.

The application-consistent path recovers to the backup’s stop LSN, so nothing recovers a transaction committed after it. There is no continuous WAL archiving (archive_mode=off, verified 2026-08-16), so there is no mechanism that could.

The exercise lost zero committed transactions — but that is a property of the exercise (the source cluster was quiescent between backup and simulated loss), not a property of the platform, and it must not be quoted as an RPO.

RPO is UNBOUNDED until you declare a schedule, and BOUNDED once you do (changed 2026-08-18, SPRINT_20260802_003 BRP-3). The recovery point is still the stop LSN of whichever backup you restore — nothing archives WAL between runs — so the RPO is exactly the declared interval plus its grace. What changed is that the interval is now a product-owned declaration rather than a private convention: Platform stores it, the cycle enforces it, and doctor.platform.backup.schedule fails when a backup that should have happened did not. An estate that declares nothing still has an unbounded RPO — and that check now says so out loud instead of leaving it to be discovered during a restore. See “Backup Schedules and Retention” below.

Backup Schedules and Retention

Status: SHIPPED 2026-08-18 (SPRINT_20260802_003 BRP-3) as a product-owned schedule, not as a stella CLI subcommand — the same shape as verify-backup.sh (SP-002): the capability ships as DevOps tooling plus product-owned state in Platform, which is how the rest of this runbook’s backup tooling ships.

What “product-owned” means here, precisely

The distinction matters, because “a script triggered by a timer” describes both the old convention and the new capability. What changed:

The cycle itself remains a host script because taking the backup is host-level work — pg_basebackup over a replication connection from inside the postgres container’s network namespace. No product service can do that: JobEngine has no process-execution surface at all, and the two containers holding a docker socket are behind opt-in compose profiles that are off by default. The product owns when and whether; the host owns how.

Declare the schedule

export STELLAOPS_PLATFORM_URL=https://stella-ops.local
export STELLAOPS_TOKEN=...   # ops.admin to declare, ops.health to read

./devops/compose/scripts/backup-schedule.sh declare \
    --interval-hours 24 --grace-hours 2 \
    --keep-last 7 --keep-days 14 --min-keep 2 \
    --min-free-gib 20 --min-free-percent 10

The storage floors are optional and default to zero, which means UNDECLARED — not “zero free is acceptable”. Declaring either one switches on platform.backup.storage-lowfor that scope; declaring neither means the product holds no opinion about the target’s headroom. Both units are offered because neither alone is honest at both ends of the scale: a percentage is meaningless on a 10 TB volume and a byte floor is meaningless on a small one. A breach of either floor is a breach.

The retention policy, stated. An artifact survives if either rule keeps it — they are a union, not an intersection:

• --keep-last N — the N newest artifacts survive.
• --keep-days D — anything younger than D days survives, even beyond N. D=0 means no age-based retention at all (not “keep everything”).
• --min-keep M — a hard floor. Pruning stops once M artifacts remain, whatever the other two rules compute.
• The newest artifact is never deleted, whatever the arithmetic says. Every other rule is arithmetic on a policy an operator can mistype, and no retention policy is worth deleting the estate’s only recovery point.

Pruning removes the artifact directory and its sibling .stellaops-backup.json sidecar and .pg_basebackup.log — they live beside the artifact rather than inside it (pg_verifybackup rejects any member the manifest omits), so retention that only removed directories would leave the backup directory filling with orphaned metadata.

Run the cycle

./devops/compose/scripts/backup-schedule.sh run

One invocation performs the whole cycle: back up (backup-postgres.sh), verify (verify-backup.sh), enforce retention against the declared policy, and report the run to Platform. Failures are reported too — a failed cycle leaves no artifact, so on disk it is indistinguishable from a cycle that never ran, and the two need different remediations.

Retention runs only after a backup verifies. Pruning around an unverified newest artifact could retire a good backup in favour of a bad one.

Install the timer

./devops/compose/scripts/backup-schedule.sh print-timer

The systemd unit pair is generated from the declared policy, so the timer and the expectation Doctor enforces cannot drift apart by transcription. If they ever do drift — the timer disabled, the unit removed, the host rebuilt — the check reports a missed backup, which is the whole point: the trigger is not trusted, it is verified.

Check it

curl -sS -H "Authorization: Bearer $STELLAOPS_TOKEN" \
  https://stella-ops.local/api/v1/platform/backup/schedule | jq .

or read it where operators already look — platform-web’s /doctor/platform-web/checks, which the console surfaces through Platform’s doctor aggregate. The check is doctor.platform.backup.schedule, and it reports:

Naming, not a gap. The done-criteria for this sprint named a db.backup-* check family. It was not minted: check.db.* is the database plugin’s four connection checks, “db” misnames a backup path covering PostgreSQL, Valkey and RustFS, and the existing check.storage.backup lives in a mounted plugin bundle whose only runtime loader (doctor-web) was retired 2026-08-17 — a missed-backup check placed there could never go red. The check therefore lives in the doctor-check/v1 family that is actually running. See the sprint’s Decisions & Risks.

Backup alerts (Notify)

The check above is a pull contract: it tells the truth whenever something asks, and emits nothing on its own. Alerts are the push half, delivered through the product’s own notification pipeline (notify-web + notifier-worker) rather than through Prometheus — see the non-goal note under What to watch for why.

Four event kinds, all on the notify:events stream, all from producer platform:

The first two ride the cycle’s own report, so they are announced the moment a run is recorded. The last two cannot be: a missed backup is the absence of an event, and an exhausted target is a standing condition. Platform therefore re-examines the declared schedule on a timer (default every 5 minutes, Platform:Backup:Alerts:PollInterval) and announces what it finds.

Each condition is announced once, not once per poll. Platform claims every alert in platform.backup_alerts before publishing it, keyed on the condition rather than the clock — a missed deadline keys on the deadline it blew, so the alert repeats only when the deadline moves, which can only happen after a backup succeeds and the estate lapses again. The claim is a database row, so this holds across restarts and across replicas.

Subscribing

Create a Notify rule matching the kinds and pointing at a channel:

curl -sS -X POST -H "Authorization: Bearer $STELLAOPS_TOKEN" \
  -H 'Content-Type: application/json' \
  https://stella-ops.local/api/v1/notify/rules \
  -d '{
    "name": "Backup alerts",
    "match": { "eventKinds": [
      "platform.backup.run-failed",
      "platform.backup.run-missed",
      "platform.backup.storage-low"
    ] },
    "actions": [ { "actionId": "backup-oncall", "channel": "<channel-id>" } ]
  }'

platform.backup.run-completed is deliberately left out of that example: subscribing to successes turns a daily backup into a daily notification, and an alert channel an operator learns to ignore is worse than no channel.

What to do on receipt

If Notify is not configured on the host, the publisher logs instead of discarding — failures and every alert at error level — so a stack without a Notify queue still leaves a trace an operator can find. That is a fallback, not a substitute: nobody is notified.

To silence announcements without silencing the measurement, set Platform:Backup:Alerts:Enabled=false. The check and the status endpoint keep reporting a missed backup exactly as before; only the announcement stops.

Scheduling it externally instead

If you prefer your own scheduler, that still works — but declare the policy anyway, or nothing can tell you when a run is skipped:

# 02:00 UTC daily — the product-owned cycle (backup, verify, prune, report).
0 2 * * *  BACKUP_DIR=/mnt/backups \
           /opt/stellaops/devops/compose/scripts/backup-schedule.sh run

# 03:00 UTC daily — RustFS + Valkey volumes, encrypted, to off-host media.
# (This also re-tars postgres crash-consistently; harmless, but the 02:00 artifact
# is the one to restore from.)
0 3 * * *  STELLAOPS_BACKUP_ENCRYPT=1 \
           STELLAOPS_BACKUP_ENCRYPTION_KEY_FILE=/secure/backup-kek.key \
           STELLAOPS_BACKUP_SIGNING_KEY_FILE=/secure/backup-sign.key \
           BACKUP_DIR=/mnt/backups \
           /opt/stellaops/devops/compose/scripts/backup.sh --yes

# 02:00 UTC daily — PostgreSQL, application-consistent. The interval you pick here
# IS your RPO (see "Recovery objectives" above), because the recovery point is this
# backup's stop LSN and nothing archives WAL between runs.
0 2 * * *  BACKUP_DIR=/mnt/backups \
           /opt/stellaops/devops/compose/scripts/backup-postgres.sh --yes

# 03:00 UTC daily — RustFS + Valkey volumes, encrypted, to off-host media.
# (This also re-tars postgres crash-consistently; harmless, but the 02:00 artifact
# is the one to restore from.)
0 3 * * *  STELLAOPS_BACKUP_ENCRYPT=1 \
           STELLAOPS_BACKUP_ENCRYPTION_KEY_FILE=/secure/backup-kek.key \
           STELLAOPS_BACKUP_SIGNING_KEY_FILE=/secure/backup-sign.key \
           BACKUP_DIR=/mnt/backups \
           /opt/stellaops/devops/compose/scripts/backup.sh --yes

# 04:00 UTC daily — verify last night's artifact. A backup nobody verified is a
# hope, not a backup; this is cheap enough to run every day.
0 4 * * *  /opt/stellaops/devops/compose/scripts/verify-backup.sh \
             "$(ls -d /mnt/backups/postgres-basebackup-* | tail -1)" \
           || logger -p daemon.err "StellaOps backup verification FAILED"

The recommended drill cadence is quarterly at minimum (see the drill runbook); backup-schedule.sh run verifies every artifact it produces, so the drill is the periodic proof rather than the only check.

Incident Procedures

INC-001: Backup script fails

Investigation:

# Re-run in dry-run to see the exact docker run and which volumes it targets
./devops/compose/scripts/backup.sh --dry-run

# Confirm free disk for the snapshot
stella doctor run --check check.storage.diskspace

Resolution by exit code (from backup.sh):

NOT IMPLEMENTED: stella backup logs, stella backup test, stella backup create --retry. There is no alert StellaBackupFailed.

INC-002: Restore failure

Symptoms: the drill exits non-zero, or services do not start after a restore.

Investigation — drill exit codes (from drill-backup-restore.sh):

Recovery path (drill runbook, “Recovery path if the drill fails”): if the drill failed between wipe and a successful restore, the stack is in the same state as a fresh-volume bootstrap. Confirm postgres is up, wait for the postgres-init scripts to finish, then docker compose … restart so services re-apply auto-migrations, and verify admin login at https://stella-ops.local.

For an encrypted artifact that will not decrypt (exit 3 from stellaops-backup-crypto): the manifest signature or ciphertext hash failed, or the wrong key was supplied. Verify you are using the same KEK and signing key files that were used at backup time.

INC-003: Backup storage full

Symptoms: a platform.backup.storage-low alert (the earliest signal — it fires while backups are still succeeding, if a storage floor is declared: see Backup alerts); backup.sh fails (often surfacing as a tar / disk-space error); or check.storage.diskspace reports a failure.

Immediate actions:

# Confirm the disk pressure
stella doctor run --check check.storage.diskspace

# Prune old artifacts in the backup directory with OS tooling, e.g.:
find "${BACKUP_DIR:-backups}" -name 'stellaops-backup-*.tar.gz*' -mtime +14 -delete

Resolution: free space or move BACKUP_DIR to a larger / off-host volume, then re-run the backup. There is no built-in cleanup/retention CLI (see “Backup Schedules and Retention” above) and no StellaBackupStorageFull alert.

Disaster Recovery Scenarios

DR-001: Complete system loss

1. Provision new infrastructure and install Stella Ops (compose stack).
2. Restore the most recent off-host snapshot:
   • decrypt it if encrypted (stellaops-backup-crypto decrypt …),
   • wipe + restore the postgres (and RustFS / Valkey) volumes from the snapshot, then docker compose … up -d.
3. Services auto-migrate on startup; verify with the doctor checks in SP-003.
4. Confirm admin login at https://stella-ops.local and update DNS / load balancer as needed.

DR-002: Database corruption

1. Stop services (docker compose … down or stop the affected services).
2. Wipe the postgres volume (wipe-postgres.sh --yes).
3. Restore from the latest known-good pg_dump (the drill’s restore phase) or the latest volume snapshot.
4. Verify data integrity by comparing row counts against the pre-loss snapshot (the drill captures authority.users, authority.clients, authority.permissions, shared.tenants; spot-check additional tenant-scoped tables such as concelier.advisories).
5. docker compose … restart and confirm admin login.

WAL-based “apply to near-corruption point” recovery is NOT IMPLEMENTED (no WAL archiving — see SP-004).

DR-003: Evidence store loss

The evidence blobs live in the RustFS volume captured by backup.sh. Restore the compose_rustfs-data volume from the snapshot, then observe evidence health via doctor:

stella doctor run --check check.evidencelocker.index       # evidence index health
stella doctor run --check check.evidencelocker.merkle      # Merkle anchor-chain integrity
stella doctor run --check check.evidencelocker.provenance  # provenance chain
stella doctor run --check check.evidencelocker.retrieval   # attestation retrieval

If the doctor checks flag drift after a restore, verify the evidence store with the real stella evidence command group (src/Cli/StellaOps.Cli/Commands/EvidenceCommandGroup.cs):

# Verify an exported evidence bundle's DSSE signatures + Rekor receipts
# (positional path to the .tar.gz; add --offline to skip Rekor in air-gap)
stella evidence verify <path-to-bundle.tar.gz>
stella evidence verify <path-to-bundle.tar.gz> --offline

Re-index and continuity have no CLI surface. The evidence reindex and evidence verify-continuity subcommands were deleted on 2026-09-03 (SPRINT_20260903_005, EVD-DEL-2): they called /api/v1/evidence/reindex/* and /api/v1/evidence/continuity/verify, which no host has ever mapped. The engine behind them (IEvidenceReindexService) still ships and is driven in-process; a post-restore recompute is therefore an engineering task, not a runbook command. The EvidenceLocker doctor checks (check.evidencelocker.index, .merkle, .provenance, .retrieval in src/Doctor/__Plugins/StellaOps.Doctor.Plugin.EvidenceLocker/Checks/) are the health-observation surface; the stella evidence subcommands are the repair surface. Run stella doctor list --category EvidenceLocker to enumerate the checks and stella evidence --help to enumerate the subcommands.

What the backups do NOT cover — signed plugin bundles

backup.sh captures three Docker volumes (compose_postgres-data, compose_rustfs-data, compose_valkey-data). The signed scanner analyzer bundles are not among them: devops/plugins/scanner/base is a read-only host bind mount into scanner-worker at /app/plugins/scanner/base (verified against the running container 2026-08-16), and host bind mounts are not volumes. Nothing in the backup path touches them.

That is correct by design — the bundles are build/release output, re-materialised by installing the release artifact, not recovered from a state backup. But it means DR-001 (“complete system loss”) is not finished when the volumes are back: the bundles must be re-installed, and their integrity is worth checking, because a scanner-worker whose bundles disagree with their own manifests rejects every analyzer plugin with contract_assembly.mismatch and silently stops scanning.

Verify with the bundle checksum sweep (checksums.sha256 vs. what is on disk across all bundles) documented in the live-deploy-operator role playbook (AGENTS.md — Memory). It is content-addressed rather than mtime- or count-based, so it catches the partial restore that a file count would wave through. Measured across 21 bundles / 4,459 entries, it cleanly caught a deliberately corrupted bundle (105 mismatched, 60 missing) and confirmed the repair.

The sweep is necessary, not sufficient — do not stop there. It proves each bundle agrees with its own manifest. A bundle can be perfectly self-coherent and still be the wrong bundle for the running image: the host↔bundle contract-assembly identity is a separate property, and it is the one that actually took this estate down. Same distinction the backup tooling draws — intact and complete is not the same as correct.

And Up (healthy) is not evidence here. During the 2026-08-16 incident the broken scanner-worker reported healthy for the entire 25-minute outage; the failure surfaced only when a scan reached execute-analyzers and every plugin was rejected with reasonCode=contract_assembly.mismatch. The proof that plugin recovery worked is a scan that completes THROUGH that stage — not a green health check, and not the absence of errors in the log. Add that to the DR-001 checklist: a restored estate whose containers are all healthy can still be scanning nothing.

On an estate run from a clone — and this one is — ANY git operation under devops/plugins/ is a live production change. The bind mount points at the working tree, so there is no deploy step between tidying the tree and changing what the estate loads. git checkout, git clean, git stash and a branch switch are all estate actions there, and they take effect instantly rather than at the next restart. Treat that path as production, not as source.

The worked example, because it is the one people reach for: git checkout -- devops/plugins/scanner/ looks like a safe revert and is not. .gitignore tracks the inventories (manifest.json, checksums.sha256, native .so/.dylib) but ignores the .dll payloads, so the checkout reverts manifests over freshly built binaries and the bundle stops matching its own checksums. Observed 2026-08-16: a ~25-minute outage in which scanner-worker rejected every analyzer plugin.

The rule that separates the two cases: failed build ⇒ restore the tree; successful build ⇒ leave it alone. A successful repack’s output is internally coherent (fresh DLLs, fresh manifests, re-signed checksums) and the tracked diff it leaves behind is expected, not damage. To recover from a genuinely bad state, re-run the packaging step rather than reverting to HEAD.

Offline / Air-Gap Backup

The shipped backup script already supports offline / air-gap operation — there is no separate create-offline / restore-offline command.

Creating an offline (encrypted) backup

STELLAOPS_BACKUP_ENCRYPT=1 \
STELLAOPS_BACKUP_ENCRYPTION_KEY_FILE=/secure/backup-kek.key \
STELLAOPS_BACKUP_SIGNING_KEY_FILE=/secure/backup-sign.key \
STELLAOPS_BACKUP_CRYPTO_BIN=/opt/stellaops/bin/stellaops-backup-crypto \
BACKUP_DIR=/media/usb/stellaops \
./devops/compose/scripts/backup.sh --yes

In air-gap deployments set STELLAOPS_BACKUP_CRYPTO_BIN to a published stellaops-backup-crypto binary so the script does not need a build server (otherwise it falls back to a published DLL in the repo, then to dotnet run from source). Key material may instead be supplied via STELLAOPS_BACKUP_ENCRYPTION_KEY / STELLAOPS_BACKUP_SIGNING_KEY env values, which the script materialises into private 0600 temp key files.

Restoring from an offline backup

# 1. Decrypt the opaque artifact back to the plaintext tar.gz (manifest verified first)
stellaops-backup-crypto decrypt \
  --in  /media/usb/stellaops/stellaops-backup-<ts>.tar.gz.stlebk \
  --out /tmp/stellaops-restore.tar.gz \
  --manifest /media/usb/stellaops/stellaops-backup-<ts>.tar.gz.stlebk.manifest.json \
  --kek-file /secure/backup-kek.key \
  --signing-key-file /secure/backup-sign.key

# 2. Restore the volumes from the decrypted tar.gz (wipe + extract + compose up), as in SP-003.

Monitoring

There is no dedicated Grafana “Backup Status” dashboard shipped in the repo. Backup health is surfaced through the stella doctor storage checks and the drill result JSON. To watch backup freshness, schedule stella doctor run --check check.storage.backup --format json and alert on a warn/fail severity, or check the latest artifacts/qa/backup-restore-drill/result-<ts>.json "status" field after each drill.

A Grafana backup dashboard is Draft/roadmap — wire one only after the backup metrics above are actually emitted by a service.

Evidence Capture

Generate a diagnostic bundle for support (the real command — there is no stella backup diagnostics):

stella doctor export -o /tmp/stellaops-diag-$(date +%Y%m%dT%H%M%S).zip

stella doctor export runs the doctor checks, packages the report plus recent logs and (unless --no-config) configuration into a ZIP, and prints a pass/warn/fail summary. Source: src/Cli/StellaOps.Cli/Commands/DoctorCommandGroup.cs.

The backup + restore drill also leaves auditable evidence under artifacts/qa/backup-restore-drill/: the pg-dump-<ts>.sql, the pre/post row-counts-*.txt, and the structured result-<ts>.json.

Escalation Path

1. L1 (On-call): Re-run failed backups (backup.sh --dry-run to diagnose), basic disk-space triage via stella doctor run --check check.storage.diskspace.
2. L2 (Platform / DevOps team): Restore drills, volume wipe + restore, schedule adjustments (host cron / systemd).
3. L3 (Architecture): Disaster-recovery execution and any PITR / WAL archiving design work (currently NOT IMPLEMENTED).

Last updated: 2026-08-18 (UTC) — SPRINT_20260802_003 BRP-4 and BRP-6: backup alerting shipped on Notify (platform.backup.run-completed / run-failed / run-missed / storage-low, a Platform-side poller, and platform.backup_alerts as the announce-once ledger). No marker was removed, and that is the honest result. The “What to watch” marker previously covered Prometheus metrics and alerts together; the alerting half shipped, the metrics half did not, so the marker was narrowed rather than retired — it now names the missing metric substrate and the Prometheus mirror of these Notify rules, which stays a declared non-goal until the observability stack deploys. Marker occurrences unchanged at 14.

BRP-6 final accounting — five distinct live gaps remain, each named deliberately: (1) PITR / WAL archiving (lines 85, 508, 1040, 1225 — one gap, restated where it bites); (2) secondary cloud object storage (151) — an intentional non-goal for an air-gap-first product, not an omission; (3) Prometheus backup metrics and the Prometheus mirror of the backup alerts (199) — needs a metric substrate and a deployed alerting stack, neither of which this sprint was authorized to create; (4) the encryption envelope on the pg_basebackup path (328) — backup.sh has one, backup-postgres.sh does not; (5) the stella CLI surface (496, 963) — stella service *, stella backup logs/test. That accounts for nine of the fourteen occurrences. The other five are not gaps: the marker legend (14, 35), two naming statements recording that something was previously flagged (394, 1072), and the historical accounting line below (1255). Sprint 20260802_003 removed exactly one marker across its whole life (BRP-3, scheduling/retention) and narrowed one (BRP-4).

Previously 2026-08-18 (UTC) — SPRINT_20260802_003 BRP-3: product-owned backup scheduling and retention shipped (backup-schedule.sh + platform.backup_policy / platform.backup_runs + the doctor.platform.backup.schedule check). Exactly one NOT IMPLEMENTED marker was retired — “Backup Schedules and Retention”, paired with that capability — taking the count from 15 to 14. Two factual corrections rode along without removing a marker: the RPO note (unbounded → bounded once a schedule is declared) and “What to watch” (which named a freshness forcing function that has had no runtime host since doctor-web’s retirement). The markers for PITR, backup metrics and alerts (BRP-4), cloud targets, the encryption envelope on the basebackup path, and the stella CLI surface all remain — those capabilities have not shipped.

Previously 2026-08-16 (UTC) — BRP-1/BRP-2/BRP-5: the application-consistent PostgreSQL backup path (SP-001A), its restore procedure and verify-backup.sh (SP-002/SP-003), and the measured recovery objectives. Two markers were amended to naming statements rather than deleted, since the CLI subcommands still do not exist.

Previously: 2026-05-31 (UTC) — reconciled against src/ and devops/; corrected the stella evidence command-group claims (DR-003) after verifying EvidenceCommandGroup.cs.