Runbook: Backup and Restore Operations

Evidence host and route currency (source-verified 2026-10-07): Evidence API references in this document target evidence-web under /api/evidence/v1. Attestation policy attestor:write accepts attest:create; attestation read/verify policies accept attest:read or attest:create, with no trusted-network bypass. Source: src/Evidence/StellaOps.Evidence.WebService/Program.cs and its application mappings. Earlier dated operational observations remain historical; verify the deployed build before applying a procedure.

Backup coverage and drill currency (source-verified 2026-10-10 at 8ae42ab238, CCP-053): the pg_dump drill drill-backup-restore.sh / .ps1 was retired. It dumped one database, always removed compose_postgres-data (the developer estate’s whole cluster volume) whichever container it was aimed at, and could report passed on two empty count files. The drill is now backup-restore-drill.md, built on backup-postgres.sh, verify-backup.sh, restore-postgres.sh and pg-census.sh. Evidence objects are not in RustFS: Evidence stores them on the filesystem volume evidence-data, and backup.sh does not archive it (see “What gets backed up”). A release-bundle installation renders its names from its .env prefixes, and the bundle ships none of the scripts in this runbook.

Current coverage (2026-10-20, PCD-557): backup.sh now includes both Evidence file volumes, requires their consumers to be quiesced, and embeds a full source/member integrity manifest. verify-backup.sh checks the complete archive through adjacent volume-backup.py; all three files and Python 3.9+ are needed. This supersedes the dated archive-omission observation above. PostgreSQL volume tars remain crash-consistent. Complete owner recovery and CA-14 still require the approved disposable drill and its application-consistent PostgreSQL companion. Source qualified at d8fc7e65d7; re-verify with python devops/compose/scripts/test_backup_evidence.py -v (fixture boundaries, including encryptor delegation; not a real cryptographic or installation proof).

Sprint: SPRINT_20260117_029_Runbook_coverage_expansion Task: RUN-004 - Backup/Restore Runbook

Purpose: Capture, verify, and restore the Stella Ops compose stack — the PostgreSQL state volume, RustFS, Valkey and both Evidence file stores — using the shipped DevOps scripts and product-native tooling. Audience: Platform / DevOps on-call performing backups, restore drills, or disaster recovery. Status: documents the shipped backup paths — an application-consistent PostgreSQL backup (backup-postgres.sh, since 2026-08-16) and the older crash-consistent volume snapshot (backup.sh), plus their restore and verification procedures; reconciled against source 2026-05-30, extended 2026-08-16 UTC. Sections marked NOT IMPLEMENTED below are roadmap, not shipped; plan recovery objectives against what this page verifies, not against the roadmap items. Measured RTO/RPO figures are in Recovery objectives — read the shape they were measured on before quoting them.

Reconciliation note (2026-05-30, re-verified 2026-05-31). This runbook was originally written against a planned stella backup … CLI surface (create / verify / restore / schedule / retention / PITR / offline). That CLI does not exist — there is no backup command group in src/Cli/StellaOps.Cli/Commands/CommandFactory.cs, no stella service lifecycle command group, and no stella db migrate subcommand (the stella db group that does exist only triggers Concelier connector jobs — fetch / merge / export). The real, shipped backup/restore mechanism is a set of DevOps shell scripts plus the product-native stellaops-backup-cryptoencryptor and the stella doctorstorage checks. Evidence-store operations are an exception: a real stella evidence command group does exist (export, verify, verify-offline, provenance, proof, store, status, card, …) — see DR-003. This document has been rewritten to match the implemented tooling. Aspirational items that are not yet implemented are explicitly marked NOT IMPLEMENTED or Draft/roadmap.

Scope

Backup and restore procedures for the Stella Ops compose stack: the PostgreSQL state volume, the RustFS object store, and Valkey. Covers volume-snapshot backups, optional product-native encryption, the disaster-recovery restore drill, and the doctor checks that watch backup health.

Canonical companion docs (read alongside this runbook):

• docs/runbooks/deploy/backup-restore-drill.md — the backup and restore drill for one named installation: cluster, file-backed stores and installation files.
• docs/modules/evidence/locker/encrypted-backup-restore.md — the encrypted backup + independent key recovery design (Sprint 20260520_088 / TOPO-115).
• docs/doctor/articles/storage/backup-directory.md — the check.storage.backup doctor check article.
• docs/runbooks/database/migration-recovery.md and docs/architecture/decisions/ADR-004-forward-only-migrations.md — the forward-only migration recovery path that postgres restore relies on.

Backup Architecture Overview

What gets backed up

The backup script (devops/compose/scripts/backup.sh) archives five required data volumes, addressed by their physical names:

For a release installation, explicitly export STELLA_VOLUME_PREFIX to resolve all five physical names. Individual PG_VOLUME, RUSTFS_VOLUME, VALKEY_VOLUME, EVIDENCE_VOLUME and PROOFCHAIN_CAS_VOLUME overrides take precedence. The helper refuses missing sources, duplicate/path-bearing selections and Evidence volumes whose Compose logical-volume labels do not match their declared role.

Not archived by backup.sh: the local timestamp authority’s CA and signer key (tsa-data), the other file-backed stores of an installation, and the installation’s files (.env, secrets/, generated keys). Losing the matching installation files loses access or signing custody; losing TSA state changes the timestamp trust root. The drill runbook lists the complete inventory and the commands for these stores (backup-restore-drill.md, sections 2, 4 and 6). PCD-557 owns the supported Evidence archive and its still-open disposable owner-recovery proof.

Before a real backup, quiesce the owning installation’s application writers in an approved window. The helper checks that neither Evidence file volume has a running consumer before and after copying and refuses an unknown writer census. It does not stop Evidence or grant a window. Keep the application writers stopped while taking backup-postgres.sh and the file archive so their database rows and objects describe the same boundary. The two observations do not prevent another operator starting a writer between them; the exclusive operation window is required.

The archive retains data/postgres, data/rustfs and data/valkey, adds data/evidence, data/proofchain-cas and backup-manifest.json, and records every member’s payload digest, type, mode and numeric ownership. The manifest identifies the selected physical sources, actual helper hashes and resolved local archive image ID. BACKUP_IMAGE selects an already present image (default alpine); the helper resolves its ID and uses --pull never --network none. It publishes no archive until copying and the final writer check succeed. Failed copies remove their pending artifact; an existing completed backup is never replaced. The integrity format admits at most 1,000,000 members and a 64 MiB manifest; larger inventories fail visibly rather than producing a partial success.

Changed 2026-08-15 — COMPOSE_PROJECT_NAME is no longer honoured here, on purpose. Every short-form volume in devops/compose/** now carries an explicit top-level name: (SPRINT_20260810_002 PTC-7 prerequisite), so volume identity is independent of the compose project name. The script previously derived ${COMPOSE_PROJECT_NAME:-compose}_postgres-data; under a project rename that derivation would have silently followed the rename onto brand-new EMPTY volumes and reported a successful backup of nothing. A hardcoded name fails loudly (exit 3, “required source volume(s) not found”); a correctly-parameterised one failed quietly.

Backup type. backup.sh produces a single crash-consistent tar.gz snapshot of the volumes. There is no continuous WAL archiving, no incremental/differential mode, and no separate per-component schedule shipped in the repo. PostgreSQL point-in-time recovery (PITR) is NOT IMPLEMENTED (see SP-004 below).

Since 2026-08-16 this is no longer the only path for PostgreSQL. backup-postgres.sh takes an application-consistent backup of the database cluster (SP-001A). Use it for PostgreSQL and keep backup.sh for RustFS and Valkey. The two consistency models are compared in the next section — the difference is not academic, and the buyer objection this runbook answers was specifically about it.

Two backup paths, and which consistency model each gives you

What “crash-consistent” actually means for the older path. It restores the way a machine restores after losing power: usually fine, never guaranteed, and never provable ahead of time. That is an acceptable model for RustFS blobs (immutable content-addressed objects) and for Valkey (cache and transport state that is rebuilt anyway). It is a weak model for a transactional database, which is why PostgreSQL now has its own path.

What the application-consistent path guarantees:

• On restore the cluster recovers to the stop LSN — the instant the copy finished. Every transaction committed at or before it is present; none after it.
• Every logical database recovers at that same point. ADR-039 puts all service databases on the shared installation server, and this is one physical copy of that server. A per-database pg_dump sweep does not give you this: each dump takes its own snapshot, so cross-service state can restore skewed.

What it does not guarantee:

• No point-in-time recovery. Recovering to an arbitrary moment needs continuous WAL archiving; archive_mode is off on this estate (verified 2026-08-16). The recovery point is the stop LSN of whichever backup you restore, so RPO is bounded by your backup interval, not by seconds (SP-004).
• It is not a correctness check on your data. A backup of a corrupted database restores that corruption faithfully.
• It does not cover RustFS or Valkey — keep taking those with backup.sh.

Worker pause and encryption (both OFF by default)

• Worker pause is optional and off by default (PAUSE_WORKERS=1 to opt in). PostgreSQL/RustFS/Valkey ship crash-consistent backups for an offline-tar snapshot; pausing live workers on a shared stack disrupts other operators.
• Encryption is optional and off by default (STELLAOPS_BACKUP_ENCRYPT=1 to opt in). When enabled, the plaintext tar.gz is run through the product-native stellaops-backup-crypto encryptor (AES-256-GCM envelope, opaque STLEBK01 artifact + HMAC-SHA256-signed manifest) and the plaintext is removed.

Storage locations

• Primary: the script writes to ./backups/ by default (override with the BACKUP_DIR environment variable, which may be an absolute path such as /var/lib/stellaops/backups/).
• Offline / air-gap: point BACKUP_DIR at removable media; enable encryption so the artifact at rest is opaque.
• Secondary cloud object storage (S3 / Azure Blob / GCS): NOT IMPLEMENTED and out of posture. Stella Ops is self-hosted / on-prem-first; there is no cloud-managed backup target wired in. Copy the local artifact to your own off-host storage with standard OS tooling.

Pre-flight Checklist

Environment verification

# Confirm the source volumes exist (dry-run prints the exact docker run it would issue)
./devops/compose/scripts/backup.sh --dry-run

# Verify backup directory accessibility + recent-backup presence (doctor check)
stella doctor run --check check.storage.backup

# Verify there is enough free disk for the snapshot
stella doctor run --check check.storage.diskspace

# Verify the database is reachable before a restore drill
stella doctor run --check check.postgres.connectivity

The check.storage.backup check reads its path from the Backup:Path or Storage:BackupPath configuration key, verifies the directory exists and is writable, scans for backup files (.bak, .backup, .tar, .tar.gz, .tgz, .zip, .sql, .dump), and warns if no backup is present or the most recent one is older than 7 days. It only runs when a backup path is configured. Source: src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Storage/Checks/BackupDirectoryCheck.cs.

What to watch

Backup health is observed in two places, and they answer different questions.

Pull — the Doctor check. The forcing function for backup freshness is doctor.platform.backup.schedule, served by platform-web’s /doctor/platform-web/checks and surfaced in the console through Platform’s doctor aggregate. It fails when a backup that should have happened did not. It answers when something asks.

Push — Notify alerts. As of BRP-4 the same facts are announced on the product’s own notification pipeline, so an operator learns about a backup gap without opening anything. See Backup alerts for the four event kinds, how to subscribe, and what to do on receipt. This is the half a pull-only check cannot cover: a backup that never happened generates no event by itself, so Platform polls the declared schedule and emits platform.backup.run-missed when the expectation is violated.

Prometheus metrics remain NOT IMPLEMENTED, and so does the Prometheus mirror of these alerts. No process emits stella_backup_last_success_timestamp, stella_backup_duration_seconds, stella_backup_size_bytes or stella_restore_test_last_success, and the alert rule names StellaBackupFailed / StellaBackupStorageFull still exist only in documentation. Keeping them named here is deliberate: they are the intended Prometheus mirror of the Notify rules below, for operators who integrate with their own alerting stack, and the rules cannot be written until there is a metric substrate and something that loads them. Verified 2026-08-16 and unchanged 2026-08-18: the telemetry overlay is not in the live estate’s config_files, it defines no alertmanager, and devops/telemetry/storage/prometheus.yaml has no rule_files stanza — so an alert rule written today could never fire. That is a declared non-goal for this sprint, not an oversight; it needs the observability stack deployed.

Corrected 2026-08-18 (BRP-3). This section previously named stella doctor run --check check.storage.backup as the freshness forcing function. That check cannot serve as one: it warns on a hardcoded seven-day threshold no policy can change, and since doctor-web was retired (2026-08-17, DOC-5 stage 4) the stellaops.doctor.storage bundle it lives in has no runtime loader — the CLI registers only the Core, Database and BinaryAnalysis plugins in-process. It is still built, signed and packaged; it is not run. Verified against src/ rather than assumed.

Events are emitted for BRP-4 to consume: every reported cycle publishes platform.backup.run-completed or platform.backup.run-failed onto the Notify event stream (best-effort — the ledger row is the record of truth, and a Notify outage never turns a good backup into a failed report). Choosing the delivery mechanism — Notify channels, a metrics path, or both — is BRP-4’s opening decision.

Standard Procedures

SP-001: Create a manual backup

When: Before upgrades, schema changes, or major configuration changes. Duration: Minutes, depending on volume size.

Arrange an approved application-writer quiesce first (drill section 4), retaining the matching backup-postgres.sh artifact and file archive. A running Evidence consumer is an exit-6 refusal even for preflight. PAUSE_WORKERS=1 only handles its explicitly selected scanner workers; it does not establish the Evidence boundary.

1. Preview the actions without touching anything:

   ./devops/compose/scripts/backup.sh --dry-run
   

2. Create a plaintext snapshot (interactive prompt; add --yes to skip):

   ./devops/compose/scripts/backup.sh --yes
   # writes ./backups/stellaops-backup-<UTC-timestamp>.tar.gz
   

3. Optional — quiesce the scanner worker during the tar (approved window only):

   PAUSE_WORKERS=1 ./devops/compose/scripts/backup.sh --yes
   

4. Optional — write an encrypted (opaque) artifact + signed manifest:

   STELLAOPS_BACKUP_ENCRYPT=1 \
   STELLAOPS_BACKUP_ENCRYPTION_KEY_FILE=/secure/backup-kek.key \
   STELLAOPS_BACKUP_SIGNING_KEY_FILE=/secure/backup-sign.key \
   ./devops/compose/scripts/backup.sh --yes
   # writes <archive>.tar.gz.stlebk + <archive>.tar.gz.stlebk.manifest.json
   # and removes the plaintext tar.gz
   

5. Custom output directory (e.g. removable media):

   BACKUP_DIR=/media/usb/stellaops ./devops/compose/scripts/backup.sh --yes
   

6. Record the final artifact’s SHA-256 (sha256sum "$ARCHIVE_FILE", with the actual produced file selected), source/image/volume inventory and operator window in independent custody. For encryption, retain the existing signed envelope manifest with its separate key-custody pointers. Embedded plaintext hashes do not authenticate an artifact rewritten together with its manifest.

The KEK and the manifest-signing key are separate keys, recoverable independently of the running stack (file or env). FileBackupKeyProvider (key files) is preferred over the env path for air-gap key escrow. See docs/modules/evidence/locker/encrypted-backup-restore.md.

SP-001A: Create an application-consistent PostgreSQL backup

When: As the primary PostgreSQL backup — before upgrades and schema changes, and on whatever cadence your RPO requires (see Recovery objectives). Duration: Roughly 40 minutes for the current live cluster; see the measured rate below.

# Preview: pre-flight only (opens a replication connection, prints the plan, copies nothing)
./devops/compose/scripts/backup-postgres.sh --dry-run

# Take the backup
./devops/compose/scripts/backup-postgres.sh --yes

# Somewhere with room, e.g. off-host media
BACKUP_DIR=/mnt/backups ./devops/compose/scripts/backup-postgres.sh --yes

It writes an artifact directory plus two siblings:

<BACKUP_DIR>/postgres-basebackup-<ts>/          <- base.tar.gz, pg_wal.tar.gz, backup_manifest
<BACKUP_DIR>/postgres-basebackup-<ts>.stellaops-backup.json   <- recovery point, sizes, SHA256s
<BACKUP_DIR>/postgres-basebackup-<ts>.pg_basebackup.log

Never put anything inside the artifact directory. pg_verifybackup fails any tar-format backup containing a member its manifest does not list — “file X is not expected in a tar format backup”. Drop a note, a checksum file or a restore record in there and every future verification of an otherwise perfect backup fails. That is why the sidecar and log are siblings. verify-backup.sh reports this case as pollution rather than corruption, because the operator response is different.

How it reaches the server without changing anything. pg_basebackup needs a replication connection, and this estate’s pg_hba.conf allows those only from 127.0.0.1 — the catch-all host all all all scram-sha-256 line does not cover replication connections, because all in the database column deliberately excludes them. Rather than edit pg_hba.conf and restart the live server, the script runs pg_basebackup in a throwaway container that joins the postgres container’s network namespace (--network container:<source>), so its 127.0.0.1 is the server’s loopback. Nothing about the running estate changes.

Load it imposes (it is a read, but not a free one):

• A full sequential read of the data directory plus a second connection streaming WAL, for the whole duration. Measured at ~37 MB/s of source on the live estate (2026-08-16). Run it in a quiet window if ingest latency matters.
• By default it requests an immediate checkpoint, a bounded write burst at the start. --spread-checkpoint is gentler but the copy does not begin until the checkpoint completes — up to several minutes of apparent hang.
• It takes no locks on user tables and blocks no writes.

Prerequisites (all satisfied on this estate; the script pre-flights them and fails with the specific remedy before copying a single byte): wal_level >= replica, max_wal_senders > 0, a backup role with REPLICATION (or superuser), and a replication line in pg_hba.conf reachable from loopback.

NOT IMPLEMENTED for this path: the encryption envelope. backup.sh can emit an opaque STLEBK01 artifact + signed manifest (SP-001 step 4); backup-postgres.sh cannot yet. Until it can, protect the artifact directory with storage-level controls, or tar-and-encrypt it out of band.

SP-002: Verify a backup

Frequency: Weekly (and before any destructive restore). Duration: Minutes.

Use verify-backup.sh — it handles both artifact shapes and fails closed:

# Application-consistent artifact: the real check
./devops/compose/scripts/verify-backup.sh backups/postgres-basebackup-<ts>

# Crash-consistent volume tar: integrity only, and it says so
./devops/compose/scripts/verify-backup.sh backups/stellaops-backup-<ts>.tar.gz

Exit codes: 0 verified, 2 artifact not found or unrecognised, 3 structural failure (missing manifest or WAL member), 4 checksum/manifest verification failed — do not rely on this backup.

For a pg_basebackup artifact it re-checks the SHA256s recorded when the backup was taken, then runs pg_verifybackupagainst postgres’s own backup_manifest, which verifies every file’s size and SHA256 and validates the manifest’s own checksum first (so a tampered manifest cannot vouch for a tampered backup).

What a pass proves, and what it does not. It proves the artifact is intact and complete: bit rot, truncation, a half-finished copy and a silently-failed transfer all fail it. It does not prove the data is semantically correct — a backup of a corrupted database verifies perfectly, because it is a faithful copy of corruption. Only the periodic restore drill answers that.

Red-proved 2026-08-16 (a verification that cannot fail is worthless), all four cases rejected: a truncated base.tar.gz; a single flipped byte with the sidecar rewritten to match — caught by pg_verifybackup naming the damaged relation, which proves the check does not depend on our own metadata; a removed pg_wal.tar.gz; and a stray file inside the artifact directory.

For volume archives the verifier reads every member, requires the exact five roots and final manifest, and checks payload hashes, type, mode and ownership. It rejects missing/extra/duplicate members, unsafe paths/links and corrupt streams. Older three-volume archives without this manifest fail as incomplete for this contract; their files are retained for their original partial-recovery use. Embedded plaintext hashes establish consistency with the recorded inventory, not authenticity against someone rewriting both data and manifest. Preserve independent artifact custody or the existing signed encryption envelope.

These hand checks alone do not verify the new inventory or Evidence recovery:

gzip -t backups/stellaops-backup-<ts>.tar.gz   # integrity of the gzip stream
tar tzf backups/stellaops-backup-<ts>.tar.gz | head   # an index sample, not coverage verification

Encrypted artifacts are verified by attempting a decrypt — the manifest HMAC signature and the ciphertext SHA-256 are checked before any plaintext is produced, and the tool fails closed on tamper / wrong key (exit code 3):

stellaops-backup-crypto decrypt \
  --in  backups/stellaops-backup-<ts>.tar.gz.stlebk \
  --out /tmp/verify.tar.gz \
  --manifest backups/stellaops-backup-<ts>.tar.gz.stlebk.manifest.json \
  --kek-file /secure/backup-kek.key \
  --signing-key-file /secure/backup-sign.key

The strongest verification remains the end-to-end restore drill (backup-restore-drill.md), which restores one installation into new volumes and compares every table of every database (pg-census.sh) and the content of every file-backed store — but it is no longer the only option, which was the point: a drill costs a reset window, so it does not happen often enough for “our backups are good” to mean anything between drills. verify-backup.sh is the between-drills answer.

Naming, not a gap (was previously flagged NOT IMPLEMENTED). There is still no stella backup verify CLI subcommand, nor --all-unverified / stella backup log-verification. The capability ships as devops/compose/scripts/verify-backup.sh (SPRINT_20260802_003 BRP-5), consistent with how the rest of this runbook’s backup tooling ships — as DevOps scripts rather than a stella backup command group. If the CLI surface is wanted later, it wraps this script; the verification itself is done.

SP-003: Restore from a backup

CAUTION: this is a destructive operation. Run only in an approved reset window.

The scripted PostgreSQL restore path is restore-postgres.sh, below. To rehearse a complete restore of one installation (cluster, file-backed stores and installation files) follow backup-restore-drill.md. The former pg_dump drill (drill-backup-restore.sh / .ps1) was retired on 2026-10-10: it dumped one database, removed the whole cluster volume and could report passed without comparing any row.

To restore an application-consistent artifact produced by backup-postgres.sh (the preferred PostgreSQL path):

# 1. Verify BEFORE destroying anything. If this fails you still have a running estate.
./devops/compose/scripts/verify-backup.sh backups/postgres-basebackup-<ts>

# 2. Stop the consumers of the target volume, then stage the cluster into it.
#    The script REFUSES a volume that a running container has mounted, refuses the
#    estate's live volumes without --i-am-restoring-the-live-estate, and refuses a
#    non-empty target without --force.
./devops/compose/scripts/restore-postgres.sh \
  --artifact backups/postgres-basebackup-<ts> \
  --target-volume compose_postgres-data \
  --i-am-restoring-the-live-estate --force --yes

# 3. Start postgres. Recovery happens HERE, not in step 2.
docker compose -f devops/compose/docker-compose.stella-ops.yml up -d postgres
docker logs stellaops-postgres 2>&1 | grep -E 'backup recovery|consistent recovery state|ready to accept'

Step 3’s log is the proof the restore worked, and it is worth reading rather than skipping. Expect these lines, in this order:

starting backup recovery with redo LSN <start>, checkpoint LSN <…>, on timeline ID 1
completed backup recovery with redo LSN <start> and end LSN <stop>
consistent recovery state reached at <stop>
database system is ready to accept connections

The <stop> LSN must equal the stopLsn in the artifact’s .stellaops-backup.json. Note it says backup recovery, not crash recovery — postgres found a backup_label and replayed the backup’s own WAL. A volume-tar restore cannot produce that line, and that difference is the whole reason this path exists.

Then continue with steps 4 and 5 below (service restart for HELLO replay, and the doctor verification).

To restore a volume snapshot produced by backup.sh (still the path for RustFS and Valkey, and the fallback for postgres):

These steps address the developer compose estate only: wipe-postgres.sh always stops stellaops-postgres and removes compose_postgres-data. For any other installation, follow backup-restore-drill.md, section 6.

1. Stop the stack:

   docker compose -f devops/compose/docker-compose.stella-ops.yml down
   

2. Wipe the existing postgres volume (canonical fresh-DB step — stops the container before docker volume rm so the removal cannot be soft-skipped):

   ./devops/compose/scripts/wipe-postgres.sh --yes
   

3. Restore the volume contents from the snapshot (decrypt first if encrypted), then bring the stack back up. Services auto-migrate their own schemas on startup (CLAUDE.md §2.7), so no manual migration step is required:

   docker compose -f devops/compose/docker-compose.stella-ops.yml up -d
   

4. After a gateway restart, restart all services so they re-send their HELLO via Valkey (transport metadata replay):

   docker compose -f devops/compose/docker-compose.stella-ops.yml restart
   

5. Verify restoration:

   stella doctor run --check check.postgres.connectivity
   stella doctor run --check check.postgres.migrations
   # then confirm admin login through https://stella-ops.local
   

NOT IMPLEMENTED: stella service stop/start/restart/health and stella db migrate. Use docker compose for lifecycle and the auto-migration-on-startup invariant instead of a manual migrate step. Amended 2026-08-16: stella backup restore still does not exist as a CLI subcommand, but the restore capability is no longer manual — it ships as devops/compose/scripts/restore-postgres.sh (SPRINT_20260802_003), with the structural safety guards described above. Independent portable evidence verification requires the declared public trust input and qualified verifier described in DR-003; it does not verify the PostgreSQL restore itself.

SP-004: Point-in-Time Recovery (PITR)

Status: NOT IMPLEMENTED (roadmap).

There is no WAL archiving configured and no stella backup restore-pitr / wal-list command. Recovery granularity is the most recent volume snapshot (SP-001) or the stop LSN of the most recent application-consistent backup (SP-001A). For point-in-time recovery you would need to enable PostgreSQL WAL archiving out of band; that is not part of the shipped tooling. Track any requirement for PITR as a new sprint task rather than relying on this section.

Recovery objectives (measured)

These are measured numbers from an executed exercise, not estimates. Read the shape they were measured on before quoting them; anything below labelled a projection is arithmetic on a measured rate, not a measured result.

The exercise — 2026-08-16 UTC

Full loss and recovery of a scratch installation: application-consistent PostgreSQL backup taken, all three data volumes destroyed, everything restored, round-trip verified. Evidence: pruned evidence docs/implplan/_evidence/2026-08-16-sprint-20260802-003-brp1-restore-exercise.json at 45c525df94.

Installation shape it was measured on. PostgreSQL 18.1, 6 databases, 273 MiB of PGDATA, 112 user tables, 160,413 rows; RustFS 35 MB / 113 objects; Valkey 161 MB / ~2,100 keys. The databases were real data read out of the live cluster with pg_dump, so schemas, indexes and row distributions are production-shaped — only the volume is smaller. The live estate is 86.3 GiB, so do not quote the exercise RTO as the estate’s RTO; use the rate below.

Both RTO numbers are reported because they measure different things and neither is padded: 65 s is the sum of the mechanical phases; 133 s is what the operator experienced, including the gaps between three separately issued commands.

Round-trip verification after restore — all PASS:

• PostgreSQL: all 112 tables and 160,413 rows identical, whole-cluster census diff empty (pg-census.sh, which compares every user table in every database — the former pg_dump drill, retired 2026-10-10, compared four tables, and a restore that dropped every other schema would pass that).
• RustFS: 113 objects, every SHA256 unchanged.
• Valkey: the planted marker key survived. Key count moved 2,103 → 2,092, which is not restore loss — 1,366 keys carry TTLs and expired across the window. Valkey holds cache and transport state, so an exact key-count round-trip is not a valid restore assertion for it; a marker key is.
• Recovery point: the cluster reached consistent recovery state at 0/14000158 — byte-identical to the stop LSN recorded before the volumes were destroyed. That is a direct proof of the consistency model, not an inference from it.

That exercise proved the data came back. It could not prove the estate came back usable, because it restored storage only — postgres, RustFS and Valkey, with no service running against them. The application-layer half is the exercise below.

The application-layer exercise — 2026-08-17 UTC

Same loss and recovery, but the scratch installation included the services and the data included real evidence capsules, so the question it answers is not “did the bytes come back” but “does the estate still do its job”. Evidence: pruned evidence docs/implplan/_evidence/2026-08-17-sprint-20260802-003-brp1-capsule-replay.json at 733039ac50.

Installation shape. PostgreSQL 18.1, 3 databases (postgres, stellaops_authority, stellaops_platform), 277 MiB of PGDATA, 97 user tables, 155,276 rows, plus the evidence object store. Running against it: authority and evidence-locker-web, both on the images the live estate runs and both configured by cloning the live containers’ configuration, so they ran the production shape (LocalHarness=false, signing on, timestamping required, Router on) rather than a relaxed harness. The data was real: four sealed decision capsules and the evidence bundles, exported read-only from the live cluster.

Read the 55 s carefully: 48 s of it was restore-postgres.sh’s running-consumer guard, not the restore. On this host the guard inspected all 51 running containers one at a time; the extraction itself took 2 s. That was fixed in the same change (the guard now asks the engine once, docker ps --filter volume=…: 0.4 s against 31.8 s, same answer, and still red-proved to refuse a volume with a live consumer). Expect the guard-inflated figure on any pre-fix copy of the script.

The capsule criterion — what was actually proven. The four capsules were replayed and DSSE-verified through evidence-locker-web twice: once before the backup, once after the total-loss restore. The replay responses are byte-identical across the two runs (SHA256 1c1e141c…), and the surviving sealed capsules verified with signatureValid: true against key id evidence-locker-capsule-key (ED25519) after the restore.

Two details keep that from being a self-congratulating check:

• The endpoint recomputes. POST /api/evidence/v1/evidence/capsules/{id}/replay re-canonicalises the stored manifest and recomputes the content hash. Flipping a single character inside a restored capsule made it return contentHashMatches:false immediately; flipping it back returned true. The result reports a content-hash comparison over the stored snapshot. It reconstructs the recorded verdict; it does not re-evaluate current Policy or re-fetch current facts. Inspect the result fields: HTTP 200 alone is not an integrity pass.
• The dataset contains capsules that legitimately fail. Two of the four are hand-seeded demo rows carrying a placeholder DSSE signature, and they replay false — before the backup and after the restore alike. So the honest expectation after a restore is not “everything green”, it is “identical to the baseline, including the reds” — which is what a restore that silently regenerated or dropped rows could not produce.

Replaying a capsule needs a real token; plan for it. The capsule routes are tenant-scoped, and the internal bypass network is Strict: it substitutes for a missing identity but grants no tenant, and X-StellaOps-TenantId is honoured only for a global-admin principal. A tokenless call is therefore refused with 400 tenant_missing. A recovery drill that intends to verify evidence must restore Authority first and mint a token — and note that OpenIddict rejects every plain-HTTP token request with This server only accepts HTTPS requests (ID2083) regardless of RequireHttpsMetadata, so call the HTTPS listener.

Live-estate scale anchor — measured 2026-08-16 UTC

Rather than estimate the 86.3 GiB estate from a 273 MiB exercise, a bounded application-consistent read was run against the live cluster and then stopped:

The 40-minute and 17 GiB figures are projections from that measured rate, not a measured full run. A full production backup was deliberately not taken in that window: the vulnerabilities-worker was mid-ingest and a 40-minute sustained read plus ~17 GiB of writes was not worth imposing for a number the sample already characterises. The read was verified harmless — postgres stayed healthy, the worker was not interrupted, and no replication slot was left behind.

Projected live RTO for PostgreSQL is dominated by extract throughput, measured at roughly 5 MiB/s of artifact through the decompress-and-write path (30 MB in 58 s). Against a ~17 GiB artifact that is on the order of an hour, plus seconds of WAL recovery. This is an extrapolation and has not been measured at live scale — treat it as a planning figure and measure it in your own reset window.

RPO — state this one honestly

RPO equals the age of the newest verified backup at the moment of loss.

The application-consistent path recovers to the backup’s stop LSN, so nothing recovers a transaction committed after it. There is no continuous WAL archiving (archive_mode=off, verified 2026-08-16), so there is no mechanism that could.

The exercise lost zero committed transactions — but that is a property of the exercise (the source cluster was quiescent between backup and simulated loss), not a property of the platform, and it must not be quoted as an RPO.

RPO is UNBOUNDED until you declare a schedule, and BOUNDED once you do (changed 2026-08-18, SPRINT_20260802_003 BRP-3). The recovery point is still the stop LSN of whichever backup you restore — nothing archives WAL between runs — so the RPO is exactly the declared interval plus its grace. What changed is that the interval is now a product-owned declaration rather than a private convention: Platform stores it, the cycle enforces it, and doctor.platform.backup.schedule fails when a backup that should have happened did not. An estate that declares nothing still has an unbounded RPO — and that check now says so out loud instead of leaving it to be discovered during a restore. See “Backup Schedules and Retention” below.

Backup Schedules and Retention

Status: SHIPPED 2026-08-18 (SPRINT_20260802_003 BRP-3) as a product-owned schedule, not as a stella CLI subcommand — the same shape as verify-backup.sh (SP-002): the capability ships as DevOps tooling plus product-owned state in Platform, which is how the rest of this runbook’s backup tooling ships.

What “product-owned” means here, precisely

The distinction matters, because “a script triggered by a timer” describes both the old convention and the new capability. What changed:

The cycle itself remains a host script because taking the backup is host-level work — pg_basebackup over a replication connection from inside the postgres container’s network namespace. No product service can do that: JobEngine has no process-execution surface at all, and the two containers holding a docker socket are behind opt-in compose profiles that are off by default. The product owns when and whether; the host owns how.

Declare the schedule

export STELLAOPS_PLATFORM_URL=https://stella-ops.local
export STELLAOPS_TOKEN=...   # ops.admin to declare, ops.health to read

./devops/compose/scripts/backup-schedule.sh declare \
    --interval-hours 24 --grace-hours 2 \
    --keep-last 7 --keep-days 14 --min-keep 2 \
    --min-free-gib 20 --min-free-percent 10

The storage floors are optional and default to zero, which means UNDECLARED — not “zero free is acceptable”. Declaring either one switches on platform.backup.storage-lowfor that scope; declaring neither means the product holds no opinion about the target’s headroom. Both units are offered because neither alone is honest at both ends of the scale: a percentage is meaningless on a 10 TB volume and a byte floor is meaningless on a small one. A breach of either floor is a breach.

The retention policy, stated. An artifact survives if either rule keeps it — they are a union, not an intersection:

• --keep-last N — the N newest artifacts survive.
• --keep-days D — anything younger than D days survives, even beyond N. D=0 means no age-based retention at all (not “keep everything”).
• --min-keep M — a hard floor. Pruning stops once M artifacts remain, whatever the other two rules compute.
• The newest artifact is never deleted, whatever the arithmetic says. Every other rule is arithmetic on a policy an operator can mistype, and no retention policy is worth deleting the estate’s only recovery point.

Pruning removes the artifact directory and its sibling .stellaops-backup.json sidecar and .pg_basebackup.log — they live beside the artifact rather than inside it (pg_verifybackup rejects any member the manifest omits), so retention that only removed directories would leave the backup directory filling with orphaned metadata.

Run the cycle

./devops/compose/scripts/backup-schedule.sh run

One invocation performs the whole cycle: back up (backup-postgres.sh), verify (verify-backup.sh), enforce retention against the declared policy, and report the run to Platform. Failures are reported too — a failed cycle leaves no artifact, so on disk it is indistinguishable from a cycle that never ran, and the two need different remediations.

Retention runs only after a backup verifies. Pruning around an unverified newest artifact could retire a good backup in favour of a bad one.

Install the timer

./devops/compose/scripts/backup-schedule.sh print-timer

The systemd unit pair is generated from the declared policy, so the timer and the expectation Doctor enforces cannot drift apart by transcription. If they ever do drift — the timer disabled, the unit removed, the host rebuilt — the check reports a missed backup, which is the whole point: the trigger is not trusted, it is verified.

Check it

curl -sS -H "Authorization: Bearer $STELLAOPS_TOKEN" \
  https://stella-ops.local/api/v1/platform/backup/schedule | jq .

or read it where operators already look — platform-web’s /doctor/platform-web/checks, which the console surfaces through Platform’s doctor aggregate. The check is doctor.platform.backup.schedule, and it reports:

Naming, not a gap. The done-criteria for this sprint named a db.backup-* check family. It was not minted: check.db.* is the database plugin’s four connection checks, “db” misnames a backup path covering PostgreSQL, Valkey and RustFS, and the existing check.storage.backup lives in a mounted plugin bundle whose only runtime loader (doctor-web) was retired 2026-08-17 — a missed-backup check placed there could never go red. The check therefore lives in the doctor-check/v1 family that is actually running. See the sprint’s Decisions & Risks.

Backup alerts (Notify)

The check above is a pull contract: it tells the truth whenever something asks, and emits nothing on its own. Alerts are the push half, delivered through the product’s own notification pipeline (notify-web + notifier-worker) rather than through Prometheus — see the non-goal note under What to watch for why.

Four event kinds, all on the notify:events stream, all from producer platform:

The first two ride the cycle’s own report, so they are announced the moment a run is recorded. The last two cannot be: a missed backup is the absence of an event, and an exhausted target is a standing condition. Platform therefore re-examines the declared schedule on a timer (default every 5 minutes, Platform:Backup:Alerts:PollInterval) and announces what it finds.

Each condition is announced once, not once per poll. Platform claims every alert in platform.backup_alerts before publishing it, keyed on the condition rather than the clock — a missed deadline keys on the deadline it blew, so the alert repeats only when the deadline moves, which can only happen after a backup succeeds and the estate lapses again. The claim is a database row, so this holds across restarts and across replicas.

Subscribing

Create a Notify rule matching the kinds and pointing at a channel:

curl -sS -X POST -H "Authorization: Bearer $STELLAOPS_TOKEN" \
  -H 'Content-Type: application/json' \
  https://stella-ops.local/api/v1/notify/rules \
  -d '{
    "name": "Backup alerts",
    "match": { "eventKinds": [
      "platform.backup.run-failed",
      "platform.backup.run-missed",
      "platform.backup.storage-low"
    ] },
    "actions": [ { "actionId": "backup-oncall", "channel": "<channel-id>" } ]
  }'

platform.backup.run-completed is deliberately left out of that example: subscribing to successes turns a daily backup into a daily notification, and an alert channel an operator learns to ignore is worse than no channel.

What to do on receipt

If Notify is not configured on the host, the publisher logs instead of discarding — failures and every alert at error level — so a stack without a Notify queue still leaves a trace an operator can find. That is a fallback, not a substitute: nobody is notified.

To silence announcements without silencing the measurement, set Platform:Backup:Alerts:Enabled=false. The check and the status endpoint keep reporting a missed backup exactly as before; only the announcement stops.

Scheduling it externally instead

If you prefer your own scheduler, that still works — but declare the policy anyway, or nothing can tell you when a run is skipped:

# 02:00 UTC daily — the product-owned cycle (backup, verify, prune, report).
0 2 * * *  BACKUP_DIR=/mnt/backups \
           /opt/stellaops/devops/compose/scripts/backup-schedule.sh run

Run backup.sh as part of the approved application-quiescence window, with the matching PostgreSQL artifact from that boundary. It now includes both Evidence file stores and refuses running consumers; installing an unattended live-volume cron job would produce refusals. The PostgreSQL schedule alone measures the database recovery point, not a complete Evidence recovery set.

# 02:00 UTC daily — PostgreSQL, application-consistent. The interval you pick here
# IS your RPO (see "Recovery objectives" above), because the recovery point is this
# backup's stop LSN and nothing archives WAL between runs.
0 2 * * *  BACKUP_DIR=/mnt/backups \
           /opt/stellaops/devops/compose/scripts/backup-postgres.sh --yes

# 03:00 UTC daily — RustFS + Valkey volumes, encrypted, to off-host media.
# (This also re-tars postgres crash-consistently; harmless, but the 02:00 artifact
# is the one to restore from.)
0 3 * * *  STELLAOPS_BACKUP_ENCRYPT=1 \
           STELLAOPS_BACKUP_ENCRYPTION_KEY_FILE=/secure/backup-kek.key \
           STELLAOPS_BACKUP_SIGNING_KEY_FILE=/secure/backup-sign.key \
           BACKUP_DIR=/mnt/backups \
           /opt/stellaops/devops/compose/scripts/backup.sh --yes

# 04:00 UTC daily — verify last night's artifact. A backup nobody verified is a
# hope, not a backup; this is cheap enough to run every day.
0 4 * * *  /opt/stellaops/devops/compose/scripts/verify-backup.sh \
             "$(ls -d /mnt/backups/postgres-basebackup-* | tail -1)" \
           || logger -p daemon.err "StellaOps backup verification FAILED"

The recommended drill cadence is quarterly at minimum (see the drill runbook); backup-schedule.sh run verifies every artifact it produces, so the drill is the periodic proof rather than the only check.

Incident Procedures

INC-001: Backup script fails

Investigation:

# Re-run in dry-run to see the exact docker run and which volumes it targets
./devops/compose/scripts/backup.sh --dry-run

# Confirm free disk for the snapshot
stella doctor run --check check.storage.diskspace

Resolution by exit code (from backup.sh):

A Docker source failure may propagate its nonzero status (including after a valid tar prefix); the pending artifact is removed and no finalized backup is claimed.

NOT IMPLEMENTED: stella backup logs, stella backup test, stella backup create --retry. There is no alert StellaBackupFailed.

INC-002: Restore failure

Symptoms: verify-backup.sh or restore-postgres.sh exits non-zero, postgres does not reach a consistent recovery state, the census differs, or services do not start after a restore.

Investigation — exit codes (from the script headers):

Recovery path: after exit 4 from restore-postgres.sh, do not start postgres on that volume; run the restore again with --force from a verified artifact. If postgres starts but its log shows crash recovery instead of backup recovery, or no consistent recovery state reached line, the volume did not receive the artifact’s backup_label; stop postgres and restore again. A non-empty pg-census.sh diff is a restore defect: keep the volume stopped and the census files, and report both. The full sequence for one installation is in backup-restore-drill.md, section 6.

For an encrypted artifact that will not decrypt (exit 3 from stellaops-backup-crypto): the manifest signature or ciphertext hash failed, or the wrong key was supplied. Verify you are using the same KEK and signing key files that were used at backup time.

INC-003: Backup storage full

Symptoms: a platform.backup.storage-low alert (the earliest signal — it fires while backups are still succeeding, if a storage floor is declared: see Backup alerts); backup.sh fails (often surfacing as a tar / disk-space error); or check.storage.diskspace reports a failure.

Immediate actions:

# Confirm the disk pressure
stella doctor run --check check.storage.diskspace

# Prune old artifacts in the backup directory with OS tooling, e.g.:
find "${BACKUP_DIR:-backups}" -name 'stellaops-backup-*.tar.gz*' -mtime +14 -delete

Resolution: free space or move BACKUP_DIR to a larger / off-host volume, then re-run the backup. There is no built-in cleanup/retention CLI (see “Backup Schedules and Retention” above) and no StellaBackupStorageFull alert.

Disaster Recovery Scenarios

DR-001: Complete system loss

1. Provision new infrastructure with Docker and the Compose plugin.
2. Restore the installation files first, to the same absolute path (for a release-bundle installation, the installation directory with .env, secrets/ and the generated keys). Do not run a fresh install instead: it generates new database passwords and keys, and the restored cluster keeps the roles and passwords of the backup.
3. Decrypt the off-host backup set if it is encrypted (stellaops-backup-crypto decrypt …), then restore the cluster and every file-backed store as in backup-restore-drill.md, section 6.
4. Services auto-migrate on startup; verify with the doctor checks in SP-003 and the evidence checks in DR-003.
5. Confirm administrator sign-in and update DNS / load balancer as needed.

DR-002: Database corruption

1. Stop every service that uses the database, then postgres.
2. Verify the most recent application-consistent artifact: verify-backup.sh <dir>/postgres-basebackup-<ts>.
3. Restore it over the damaged volume. --force clears the volume first; the developer estate’s volumes also need --i-am-restoring-the-live-estate: restore-postgres.sh --artifact <dir>/postgres-basebackup-<ts> --target-volume <volume> --force --yes.
4. Start postgres alone and check the recovery log against the artifact’s stopLsn (SP-003). Every transaction committed after that LSN is lost (SP-004).
5. If a census was taken with the backup, compare pg-census.sh output against it.
6. Start the services and confirm administrator sign-in.

WAL-based “apply to near-corruption point” recovery is NOT IMPLEMENTED (no WAL archiving — see SP-004).

DR-003: Evidence store loss

Source correction (2026-10-07): the consolidated Evidence overlay configures filesystem storage at /data/evidence and proof-chain CAS at /app/data/proofchain-cas, mounted from compose_evidence-data and compose_attestor-proofchain-cas. The release bundle uses its declared volume prefix. Inspect the deployed manifests to resolve actual volume identities. Current backup.sh includes both Evidence volumes as distinct roots with the byte/ownership inventory described above; RustFS remains a different store. Preserve the matching application-consistent PostgreSQL backup from the same writer quiesce. PCD-557’s source archive controls do not establish its still-required real recovered-subject proof.

After verifying the archive and the declared recovery set, restore each file root only into its inventoried, empty, stopped target volume. For example, with ROLE=evidence or ROLE=proofchain-cas, use the corresponding recorded physical TARGET_VOLUME, a trusted absolute ARCHIVE_DIR, its ARCHIVE_NAME and the installation’s retained digest-pinned RESTORE_IMAGE, whose tar must support --numeric-owner. A mismatch or any running consumer stops this procedure. Do not substitute a raw PostgreSQL volume tar for the application-consistent PostgreSQL recovery described in SP-003.

# Approved disposable/recovery target only; confirm these values against its inventory.
case "$ROLE" in evidence|proofchain-cas) ;; *) echo 'Unexpected Evidence archive root' >&2; exit 3 ;; esac
docker volume inspect "$TARGET_VOLUME" || exit 3
CONSUMERS="$(docker ps --filter "volume=$TARGET_VOLUME" --format '{{.ID}}')" || exit 3
test -z "$CONSUMERS" || { echo 'Target has running consumers' >&2; exit 3; }
docker run --rm --network none -v "$TARGET_VOLUME:/inspect:ro" "$RESTORE_IMAGE" \
  sh -ec 'test -z "$(ls -A /inspect)"' || exit 3              # refuse a non-empty target
docker run --rm --network none -v "$ARCHIVE_DIR:/backup:ro" \
  -v "$TARGET_VOLUME:/restore/data/$ROLE" "$RESTORE_IMAGE" \
  tar --numeric-owner -xzf "/backup/$ARCHIVE_NAME" -C /restore "data/$ROLE"

Mounting the target at its original archive subtree restores that volume root’s numeric ownership and mode as well as its descendants. On Git Bash export MSYS_NO_PATHCONV=1 and MSYS2_ARG_CONV_EXCL='*' before these container commands. Compare frozen object/CAS bytes and sidecars before starting writers or making a materializing read: the native download route can rebuild a missing package, and the portable route can create a successor row. Their HTTP success alone cannot prove that the original archived object survived.

After an authorized restore, observe the current owner’s database checks:

curl -fsS -H "Authorization: Bearer $TOKEN" "$EVIDENCE_URL/doctor/evidence/checks"

This source-mapped surface requires ops.health; inspect the returned service, check identities, unhealthy results and evidence. Readiness alone cannot prove the restored artifacts or proof chain. The old check.evidencelocker.* filesystem plugin is not this host’s doctor composition, and its HTTP retrieval appends an absent sample route. Do not use those predecessor checks as the restore verdict.

After restore, compare exact owner retrieval and known-good/known-failing evidence with the retained baseline. Independently verify a portable release export using a qualified packaged CLI carrying CCP-038 and an independently declared public JWKS or SPKI trust file (src/Cli/StellaOps.Cli/Commands/EvidenceCommandGroup.cs):

stella evidence verify-offline <path-to-portable-bundle.tgz> --trust-root <path-to-declared-public-jwks-or-spki> -o json

The command checks proof presence, checksums, manifest bindings, signature and signed identity without network access; exit 0 requires trusted verification. Missing trust fails closed. Source availability does not prove the packaged CLI or trust input is available on the recovery host. The legacy archive inspector reports cryptographic authenticity as unverified and is not the recovery verdict. Capsule replay separately reconstructs the stored sealed manifest and recomputes its content hash; it is not a fresh Policy evaluation or a signature check. Neither operation proves complete backup coverage or PostgreSQL restore consistency.

Re-index and continuity have no CLI surface. The evidence reindex and evidence verify-continuity subcommands were deleted on 2026-09-03 (SPRINT_20260903_005, EVD-DEL-2): they called /api/v1/evidence/reindex/* and /api/v1/evidence/continuity/verify, which no host has ever mapped. The engine behind them (IEvidenceReindexService) still ships and is driven in-process; a post-restore recompute is therefore an engineering task, not a runbook command. Use the consolidated owner’s GET /doctor/evidence/checks for its registered observations and the mapped bundle/capsule APIs for their specific evidence operations. A documentation correction does not establish a complete backup or recovery drill.

What the backups do NOT cover — signed plugin bundles

backup.sh captures five Docker volumes (the three infrastructure sources and the two Evidence file stores listed above). The signed scanner analyzer bundles are not among them: .runtime/plugins/scanner/base is a read-only host bind mount into scanner-worker at /app/plugins/scanner/base (verified against the running container 2026-08-16, then at devops/plugins/scanner/base), and host bind mounts are not volumes. Nothing in the backup path touches them.

That is correct by design — the bundles are build/release output, re-materialised by installing the release artifact, not recovered from a state backup. But it means DR-001 (“complete system loss”) is not finished when the volumes are back: the bundles must be re-installed, and their integrity is worth checking, because a scanner-worker whose bundles disagree with their own manifests rejects every analyzer plugin with contract_assembly.mismatch and silently stops scanning.

Verify with the bundle checksum sweep (checksums.sha256 vs. what is on disk across all bundles) documented in the live-deploy-operator role playbook (an internal agent playbook, not published in this documentation). It is content-addressed rather than mtime- or count-based, so it catches the partial restore that a file count would wave through. Measured across 21 bundles / 4,459 entries, it cleanly caught a deliberately corrupted bundle (105 mismatched, 60 missing) and confirmed the repair.

The sweep is necessary, not sufficient — do not stop there. It proves each bundle agrees with its own manifest. A bundle can be perfectly self-coherent and still be the wrong bundle for the running image: the host↔bundle contract-assembly identity is a separate property, and it is the one that actually took this estate down. Same distinction the backup tooling draws — intact and complete is not the same as correct.

And Up (healthy) is not evidence here. During the 2026-08-16 incident the broken scanner-worker reported healthy for the entire 25-minute outage; the failure surfaced only when a scan reached execute-analyzers and every plugin was rejected with reasonCode=contract_assembly.mismatch. The proof that plugin recovery worked is a scan that completes THROUGH that stage — not a green health check, and not the absence of errors in the log. Add that to the DR-001 checklist: a restored estate whose containers are all healthy can still be scanning nothing.

On an estate run from a clone — and this one is — ANY write under .runtime/plugins/ is a live production change. The bind mount points at the primary tree’s git-ignored .runtime/plugins/, so there is no deploy step between a packaging run and changing what the estate loads. A packaging run and git clean -x are estate actions there, and they take effect instantly rather than at the next restart. Treat that path as production, not as scratch.

The worked example dates from before the bundles left the tracked devops/plugins/ tree: git checkout -- devops/plugins/scanner/ looked like a safe revert and was not. .gitignore tracked the inventories (manifest.json, checksums.sha256, native .so/.dylib) but ignored the .dll payloads, so the checkout reverted manifests over freshly built binaries and the bundle stopped matching its own checksums. Observed 2026-08-16: a ~25-minute outage in which scanner-worker rejected every analyzer plugin.

Recovery rule: re-run the packaging step. .runtime/plugins/ has no git copy to restore. A successful repack’s output is internally coherent (fresh DLLs, fresh manifests, re-signed checksums); a failed or partial one is replaced by the next successful run, never by reverting to HEAD.

Offline / Air-Gap Backup

The shipped backup script already supports offline / air-gap operation — there is no separate create-offline / restore-offline command.

Creating an offline (encrypted) backup

STELLAOPS_BACKUP_ENCRYPT=1 \
STELLAOPS_BACKUP_ENCRYPTION_KEY_FILE=/secure/backup-kek.key \
STELLAOPS_BACKUP_SIGNING_KEY_FILE=/secure/backup-sign.key \
STELLAOPS_BACKUP_CRYPTO_BIN=/opt/stellaops/bin/stellaops-backup-crypto \
BACKUP_DIR=/media/usb/stellaops \
./devops/compose/scripts/backup.sh --yes

In air-gap deployments set STELLAOPS_BACKUP_CRYPTO_BIN to a published stellaops-backup-crypto binary so the script does not need a build server (otherwise it falls back to a published DLL in the repo, then to dotnet run from source). Key material may instead be supplied via STELLAOPS_BACKUP_ENCRYPTION_KEY / STELLAOPS_BACKUP_SIGNING_KEY env values, which the script materialises into private 0600 temp key files.

Restoring from an offline backup

# 1. Decrypt the opaque artifact back to the plaintext tar.gz (manifest verified first)
stellaops-backup-crypto decrypt \
  --in  /media/usb/stellaops/stellaops-backup-<ts>.tar.gz.stlebk \
  --out /tmp/stellaops-restore.tar.gz \
  --manifest /media/usb/stellaops/stellaops-backup-<ts>.tar.gz.stlebk.manifest.json \
  --kek-file /secure/backup-kek.key \
  --signing-key-file /secure/backup-sign.key

# 2. Restore the volumes from the decrypted tar.gz (wipe + extract + compose up), as in SP-003.

Monitoring

There is no dedicated Grafana “Backup Status” dashboard shipped in the repo. Backup health is surfaced through the stella doctor storage checks and the product-owned backup cycle’s alerts (see “Backup alerts (Notify)”). To watch backup freshness, schedule stella doctor run --check check.storage.backup --format json and alert on a warn/fail severity. A drill leaves its own record set (backup-restore-drill.md, section 8).

A Grafana backup dashboard is Draft/roadmap — wire one only after the backup metrics above are actually emitted by a service.

Evidence Capture

Generate a diagnostic bundle for support (the real command — there is no stella backup diagnostics):

stella doctor export -o /tmp/stellaops-diag-$(date +%Y%m%dT%H%M%S).zip

stella doctor export runs the doctor checks, packages the report plus recent logs and (unless --no-config) configuration into a ZIP, and prints a pass/warn/fail summary. Source: src/Cli/StellaOps.Cli/Commands/DoctorCommandGroup.cs.

The backup and restore tools leave auditable records beside each artifact: the .stellaops-backup.json sidecar with the stop LSN (backup-postgres.sh) and the .restore-<volume>-<ts>.json record (restore-postgres.sh). A drill adds its census files, file-store checksum lists and measurement record (backup-restore-drill.md, sections 3, 4, 6 and 8).

Escalation Path

1. L1 (On-call): Re-run failed backups (backup.sh --dry-run to diagnose), basic disk-space triage via stella doctor run --check check.storage.diskspace.
2. L2 (Platform / DevOps team): Restore drills, volume wipe + restore, schedule adjustments (host cron / systemd).
3. L3 (Architecture): Disaster-recovery execution and any PITR / WAL archiving design work (currently NOT IMPLEMENTED).

Last updated: 2026-08-18 (UTC) — SPRINT_20260802_003 BRP-4 and BRP-6: backup alerting shipped on Notify (platform.backup.run-completed / run-failed / run-missed / storage-low, a Platform-side poller, and platform.backup_alerts as the announce-once ledger). No marker was removed, and that is the honest result. The “What to watch” marker previously covered Prometheus metrics and alerts together; the alerting half shipped, the metrics half did not, so the marker was narrowed rather than retired — it now names the missing metric substrate and the Prometheus mirror of these Notify rules, which stays a declared non-goal until the observability stack deploys. Marker occurrences unchanged at 14.

BRP-6 final accounting — five distinct live gaps remain, each named deliberately: (1) PITR / WAL archiving (lines 85, 508, 1040, 1225 — one gap, restated where it bites); (2) secondary cloud object storage (151) — an intentional non-goal for an air-gap-first product, not an omission; (3) Prometheus backup metrics and the Prometheus mirror of the backup alerts (199) — needs a metric substrate and a deployed alerting stack, neither of which this sprint was authorized to create; (4) the encryption envelope on the pg_basebackup path (328) — backup.sh has one, backup-postgres.sh does not; (5) the stella CLI surface (496, 963) — stella service *, stella backup logs/test. That accounts for nine of the fourteen occurrences. The other five are not gaps: the marker legend (14, 35), two naming statements recording that something was previously flagged (394, 1072), and the historical accounting line below (1255). Sprint 20260802_003 removed exactly one marker across its whole life (BRP-3, scheduling/retention) and narrowed one (BRP-4).

Previously 2026-08-18 (UTC) — SPRINT_20260802_003 BRP-3: product-owned backup scheduling and retention shipped (backup-schedule.sh + platform.backup_policy / platform.backup_runs + the doctor.platform.backup.schedule check). Exactly one NOT IMPLEMENTED marker was retired — “Backup Schedules and Retention”, paired with that capability — taking the count from 15 to 14. Two factual corrections rode along without removing a marker: the RPO note (unbounded → bounded once a schedule is declared) and “What to watch” (which named a freshness forcing function that has had no runtime host since doctor-web’s retirement). The markers for PITR, backup metrics and alerts (BRP-4), cloud targets, the encryption envelope on the basebackup path, and the stella CLI surface all remain — those capabilities have not shipped.

Previously 2026-08-16 (UTC) — BRP-1/BRP-2/BRP-5: the application-consistent PostgreSQL backup path (SP-001A), its restore procedure and verify-backup.sh (SP-002/SP-003), and the measured recovery objectives. Two markers were amended to naming statements rather than deleted, since the CLI subcommands still do not exist.

Previously: 2026-05-31 (UTC) — reconciled against src/ and devops/; corrected the stella evidence command-group claims (DR-003) after verifying EvidenceCommandGroup.cs.