Runbook: Backup and Restore Operations

Sprint: SPRINT_20260117_029_Runbook_coverage_expansion Task: RUN-004 - Backup/Restore Runbook

Purpose: Capture, verify, and restore the Stella Ops compose stack — the PostgreSQL state volume, the RustFS object store, and Valkey — using the shipped DevOps scripts and product-native tooling. Audience: Platform / DevOps on-call performing backups, restore drills, or disaster recovery. Status: PRODUCTION-READY (2026-01-17 UTC); reconciled against source 2026-05-30 UTC.

Reconciliation note (2026-05-30, re-verified 2026-05-31). This runbook was originally written against a planned stella backup … CLI surface (create / verify / restore / schedule / retention / PITR / offline). That CLI does not exist — there is no backup command group in src/Cli/StellaOps.Cli/Commands/CommandFactory.cs, no stella service lifecycle command group, and no stella db migrate subcommand (the stella db group that does exist only triggers Concelier connector jobs — fetch / merge / export). The real, shipped backup/restore mechanism is a set of DevOps shell scripts plus the product-native stellaops-backup-cryptoencryptor and the stella doctorstorage checks. Evidence-store operations are an exception: a real stella evidence command group does exist (verify, reindex, verify-continuity, provenance, proof, store, status, card, …) — see DR-003. This document has been rewritten to match the implemented tooling. Aspirational items that are not yet implemented are explicitly marked NOT IMPLEMENTED or Draft/roadmap.

Scope

Backup and restore procedures for the Stella Ops compose stack: the PostgreSQL state volume, the RustFS object store, and Valkey. Covers volume-snapshot backups, optional product-native encryption, the disaster-recovery restore drill, and the doctor checks that watch backup health.

Canonical companion docs (read alongside this runbook):

• docs/runbooks/backup-restore-drill.md — the canonical pg_dump → wipe → restore drill (DEVOPS-OPS-029-01).
• docs/modules/evidence-locker/encrypted-backup-restore.md — the encrypted backup + independent key recovery design (Sprint 20260520_088 / TOPO-115).
• docs/doctor/articles/storage/backup-directory.md — the check.storage.backup doctor check article.
• docs/runbooks/migration-recovery.md and docs/architecture/decisions/ADR-004-forward-only-migrations.md — the forward-only migration recovery path that postgres restore relies on.

Backup Architecture Overview

What gets backed up

The shipped backup script (devops/compose/scripts/backup.sh) snapshots the three live data volumes derived from the compose project name (default project compose):

Override COMPOSE_PROJECT_NAME (or PG_VOLUME / RUSTFS_VOLUME / VALKEY_VOLUME) if the stack was brought up under a different project name.

Backup type. The script produces a single crash-consistent tar.gz snapshot of the volumes. There is no continuous WAL archiving, no incremental/differential mode, and no separate per-component schedule shipped in the repo. PostgreSQL point-in-time recovery (PITR) is NOT IMPLEMENTED (see SP-004 below).

Worker pause and encryption (both OFF by default)

• Worker pause is optional and off by default (PAUSE_WORKERS=1 to opt in). PostgreSQL/RustFS/Valkey ship crash-consistent backups for an offline-tar snapshot; pausing live workers on a shared stack disrupts other operators.
• Encryption is optional and off by default (STELLAOPS_BACKUP_ENCRYPT=1 to opt in). When enabled, the plaintext tar.gz is run through the product-native stellaops-backup-crypto encryptor (AES-256-GCM envelope, opaque STLEBK01 artifact + HMAC-SHA256-signed manifest) and the plaintext is removed.

Storage locations

• Primary: the script writes to ./backups/ by default (override with the BACKUP_DIR environment variable, which may be an absolute path such as /var/lib/stellaops/backups/).
• Offline / air-gap: point BACKUP_DIR at removable media; enable encryption so the artifact at rest is opaque.
• Secondary cloud object storage (S3 / Azure Blob / GCS): NOT IMPLEMENTED and out of posture. Stella Ops is self-hosted / on-prem-first; there is no cloud-managed backup target wired in. Copy the local artifact to your own off-host storage with standard OS tooling.

Pre-flight Checklist

Environment verification

# Confirm the source volumes exist (dry-run prints the exact docker run it would issue)
./devops/compose/scripts/backup.sh --dry-run

# Verify backup directory accessibility + recent-backup presence (doctor check)
stella doctor run --check check.storage.backup

# Verify there is enough free disk for the snapshot
stella doctor run --check check.storage.diskspace

# Verify the database is reachable before a restore drill
stella doctor run --check check.postgres.connectivity

The check.storage.backup check reads its path from the Backup:Path or Storage:BackupPath configuration key, verifies the directory exists and is writable, scans for backup files (.bak, .backup, .tar, .tar.gz, .tgz, .zip, .sql, .dump), and warns if no backup is present or the most recent one is older than 7 days. It only runs when a backup path is configured. Source: src/Doctor/__Plugins/StellaOps.Doctor.Plugin.Storage/Checks/BackupDirectoryCheck.cs.

What to watch

Backup health is observed through the doctor checks above, not through dedicated Prometheus metrics. The previously-documented metrics (stella_backup_last_success_timestamp, stella_backup_duration_seconds, stella_backup_size_bytes, stella_restore_test_last_success) and the alerts StellaBackupFailed / StellaBackupStorageFull are NOT IMPLEMENTED — no such metrics or alert rules exist in src/. Use stella doctor run --check check.storage.backup (and the drill result JSON, below) as the forcing function for backup freshness.

Standard Procedures

SP-001: Create a manual backup

When: Before upgrades, schema changes, or major configuration changes. Duration: Minutes, depending on volume size.

1. Preview the actions without touching anything:

   ./devops/compose/scripts/backup.sh --dry-run
   

2. Create a plaintext snapshot (interactive prompt; add --yes to skip):

   ./devops/compose/scripts/backup.sh --yes
   # writes ./backups/stellaops-backup-<UTC-timestamp>.tar.gz
   

3. Optional — quiesce the scanner worker during the tar (approved window only):

   PAUSE_WORKERS=1 ./devops/compose/scripts/backup.sh --yes
   

4. Optional — write an encrypted (opaque) artifact + signed manifest:

   STELLAOPS_BACKUP_ENCRYPT=1 \
   STELLAOPS_BACKUP_ENCRYPTION_KEY_FILE=/secure/backup-kek.key \
   STELLAOPS_BACKUP_SIGNING_KEY_FILE=/secure/backup-sign.key \
   ./devops/compose/scripts/backup.sh --yes
   # writes <archive>.tar.gz.stlebk + <archive>.tar.gz.stlebk.manifest.json
   # and removes the plaintext tar.gz
   

5. Custom output directory (e.g. removable media):

   BACKUP_DIR=/media/usb/stellaops ./devops/compose/scripts/backup.sh --yes
   

The KEK and the manifest-signing key are separate keys, recoverable independently of the running stack (file or env). FileBackupKeyProvider (key files) is preferred over the env path for air-gap key escrow. See docs/modules/evidence-locker/encrypted-backup-restore.md.

SP-002: Verify a backup

Frequency: Weekly (and before any destructive restore). Duration: Minutes.

Plaintext snapshots are plain tar.gz; verify with standard tooling:

gzip -t backups/stellaops-backup-<ts>.tar.gz   # integrity of the gzip stream
tar tzf backups/stellaops-backup-<ts>.tar.gz | head   # lists data/postgres, data/rustfs, data/valkey

Encrypted artifacts are verified by attempting a decrypt — the manifest HMAC signature and the ciphertext SHA-256 are checked before any plaintext is produced, and the tool fails closed on tamper / wrong key (exit code 3):

stellaops-backup-crypto decrypt \
  --in  backups/stellaops-backup-<ts>.tar.gz.stlebk \
  --out /tmp/verify.tar.gz \
  --manifest backups/stellaops-backup-<ts>.tar.gz.stlebk.manifest.json \
  --kek-file /secure/backup-kek.key \
  --signing-key-file /secure/backup-sign.key

The strongest verification is the end-to-end restore drill (SP-003 / the DEVOPS-OPS-029-01 drill), which restores into a clean cluster and asserts row-count round-trip.

NOT IMPLEMENTED: there is no stella backup verify, --all-unverified, or stella backup log-verification command. Verification is done with the tools above plus the drill.

SP-003: Restore from a backup

CAUTION: this is a destructive operation. Run only in an approved reset window.

The canonical, scripted restore path is the backup + restore drill, which pg_dumps the live cluster, wipes the volume, brings postgres back, and restores — asserting that representative row counts round-trip. See docs/runbooks/backup-restore-drill.md for the full operator checklist.

# Dry-run first (default mode; prints actions, touches nothing, exits 10)
./devops/compose/scripts/drill-backup-restore.sh

# Confirmed destructive run (approved reset window only)
./devops/compose/scripts/drill-backup-restore.sh --confirm

# Windows / PowerShell
.\devops\compose\scripts\drill-backup-restore.ps1 -Confirm

To restore a volume snapshot produced by backup.sh instead of a pg_dump:

1. Stop the stack:

   docker compose -f devops/compose/docker-compose.stella-ops.yml down
   

2. Wipe the existing postgres volume (canonical fresh-DB step — stops the container before docker volume rm so the removal cannot be soft-skipped):

   ./devops/compose/scripts/wipe-postgres.sh --yes
   

3. Restore the volume contents from the snapshot (decrypt first if encrypted), then bring the stack back up. Services auto-migrate their own schemas on startup (CLAUDE.md §2.7), so no manual migration step is required:

   docker compose -f devops/compose/docker-compose.stella-ops.yml up -d
   

4. After a gateway restart, restart all services so they re-send their HELLO via Valkey (transport metadata replay):

   docker compose -f devops/compose/docker-compose.stella-ops.yml restart
   

5. Verify restoration:

   stella doctor run --check check.postgres.connectivity
   stella doctor run --check check.postgres.migrations
   # then confirm admin login through https://stella-ops.local
   

NOT IMPLEMENTED: stella service stop/start/restart/health, stella backup restore, and stella db migrate. Use docker compose for lifecycle, the scripts for restore, and the auto-migration-on-startup invariant instead of a manual migrate step. (Note: stella evidence verify does exist — it verifies an exported evidence bundle’s DSSE/Rekor signatures, not the postgres restore; see DR-003.)

SP-004: Point-in-Time Recovery (PITR)

Status: NOT IMPLEMENTED (roadmap).

There is no WAL archiving configured and no stella backup restore-pitr / wal-list command. Recovery granularity is the most recent volume snapshot (SP-001) or the most recent pg_dump captured by the drill. For point-in-time recovery you would need to enable PostgreSQL WAL archiving out of band; that is not part of the shipped tooling. Track any requirement for PITR as a new sprint task rather than relying on this section.

Backup Schedules and Retention

Status: NOT IMPLEMENTED as product CLI (operator-scheduled).

There is no stella backup schedule or stella backup retention command. The shipped backup script is invoked on demand or from an external scheduler. To run backups on a cadence, wrap backup.sh in a host cron / systemd timer, e.g.:

# 02:00 UTC daily, non-interactive, encrypted, to off-host media
0 2 * * *  STELLAOPS_BACKUP_ENCRYPT=1 \
           STELLAOPS_BACKUP_ENCRYPTION_KEY_FILE=/secure/backup-kek.key \
           STELLAOPS_BACKUP_SIGNING_KEY_FILE=/secure/backup-sign.key \
           BACKUP_DIR=/mnt/backups \
           /opt/stellaops/devops/compose/scripts/backup.sh --yes

Retention is the operator’s responsibility (rotate / prune the BACKUP_DIR contents with standard OS tooling). The check.storage.backup doctor check warns when the newest backup is older than 7 days — use it as the freshness forcing function. The recommended drill cadence is quarterly at minimum (see the drill runbook).

Incident Procedures

INC-001: Backup script fails

Investigation:

# Re-run in dry-run to see the exact docker run and which volumes it targets
./devops/compose/scripts/backup.sh --dry-run

# Confirm free disk for the snapshot
stella doctor run --check check.storage.diskspace

Resolution by exit code (from backup.sh):

NOT IMPLEMENTED: stella backup logs, stella backup test, stella backup create --retry. There is no alert StellaBackupFailed.

INC-002: Restore failure

Symptoms: the drill exits non-zero, or services do not start after a restore.

Investigation — drill exit codes (from drill-backup-restore.sh):

Recovery path (drill runbook, “Recovery path if the drill fails”): if the drill failed between wipe and a successful restore, the stack is in the same state as a fresh-volume bootstrap. Confirm postgres is up, wait for the postgres-init scripts to finish, then docker compose … restart so services re-apply auto-migrations, and verify admin login at https://stella-ops.local.

For an encrypted artifact that will not decrypt (exit 3 from stellaops-backup-crypto): the manifest signature or ciphertext hash failed, or the wrong key was supplied. Verify you are using the same KEK and signing key files that were used at backup time.

INC-003: Backup storage full

Symptoms: backup.sh fails (often surfacing as a tar / disk-space error), or check.storage.diskspace reports a failure.

Immediate actions:

# Confirm the disk pressure
stella doctor run --check check.storage.diskspace

# Prune old artifacts in the backup directory with OS tooling, e.g.:
find "${BACKUP_DIR:-backups}" -name 'stellaops-backup-*.tar.gz*' -mtime +14 -delete

Resolution: free space or move BACKUP_DIR to a larger / off-host volume, then re-run the backup. There is no built-in cleanup/retention CLI (see “Backup Schedules and Retention” above) and no StellaBackupStorageFull alert.

Disaster Recovery Scenarios

DR-001: Complete system loss

1. Provision new infrastructure and install Stella Ops (compose stack).
2. Restore the most recent off-host snapshot:
   • decrypt it if encrypted (stellaops-backup-crypto decrypt …),
   • wipe + restore the postgres (and RustFS / Valkey) volumes from the snapshot, then docker compose … up -d.
3. Services auto-migrate on startup; verify with the doctor checks in SP-003.
4. Confirm admin login at https://stella-ops.local and update DNS / load balancer as needed.

DR-002: Database corruption

1. Stop services (docker compose … down or stop the affected services).
2. Wipe the postgres volume (wipe-postgres.sh --yes).
3. Restore from the latest known-good pg_dump (the drill’s restore phase) or the latest volume snapshot.
4. Verify data integrity by comparing row counts against the pre-loss snapshot (the drill captures authority.users, authority.clients, authority.permissions, shared.tenants; spot-check additional tenant-scoped tables such as concelier.advisories).
5. docker compose … restart and confirm admin login.

WAL-based “apply to near-corruption point” recovery is NOT IMPLEMENTED (no WAL archiving — see SP-004).

DR-003: Evidence store loss

The evidence blobs live in the RustFS volume captured by backup.sh. Restore the compose_rustfs-data volume from the snapshot, then observe evidence health via doctor:

stella doctor run --check check.evidencelocker.index       # evidence index health
stella doctor run --check check.evidencelocker.merkle      # Merkle anchor-chain integrity
stella doctor run --check check.evidencelocker.provenance  # provenance chain
stella doctor run --check check.evidencelocker.retrieval   # attestation retrieval

If the doctor checks flag drift after a restore, repair the evidence store with the real stella evidence command group (src/Cli/StellaOps.Cli/Commands/EvidenceCommandGroup.cs):

# Re-index evidence bundles after a restore / schema / algorithm change (dry-run first)
stella evidence reindex --dry-run
stella evidence reindex

# Verify chain-of-custody after the reindex / upgrade
stella evidence verify-continuity

# Verify an exported evidence bundle's DSSE signatures + Rekor receipts
# (positional path to the .tar.gz; add --offline to skip Rekor in air-gap)
stella evidence verify <path-to-bundle.tar.gz>
stella evidence verify <path-to-bundle.tar.gz> --offline

Naming correction (was previously flagged NOT IMPLEMENTED). The aspirational commands stella evidence index rebuild / stella evidence anchor verify do not exist under those names, but the capabilities do ship under stella evidence reindex, stella evidence verify-continuity, and stella evidence verify. The EvidenceLocker doctor checks (check.evidencelocker.index, .merkle, .provenance, .retrieval in src/Doctor/__Plugins/StellaOps.Doctor.Plugin.EvidenceLocker/Checks/) are the health-observation surface; the stella evidence subcommands are the repair surface. Run stella doctor list --category EvidenceLocker to enumerate the checks and stella evidence --help to enumerate the subcommands.

Offline / Air-Gap Backup

The shipped backup script already supports offline / air-gap operation — there is no separate create-offline / restore-offline command.

Creating an offline (encrypted) backup

STELLAOPS_BACKUP_ENCRYPT=1 \
STELLAOPS_BACKUP_ENCRYPTION_KEY_FILE=/secure/backup-kek.key \
STELLAOPS_BACKUP_SIGNING_KEY_FILE=/secure/backup-sign.key \
STELLAOPS_BACKUP_CRYPTO_BIN=/opt/stellaops/bin/stellaops-backup-crypto \
BACKUP_DIR=/media/usb/stellaops \
./devops/compose/scripts/backup.sh --yes

In air-gap deployments set STELLAOPS_BACKUP_CRYPTO_BIN to a published stellaops-backup-crypto binary so the script does not need a build server (otherwise it falls back to a published DLL in the repo, then to dotnet run from source). Key material may instead be supplied via STELLAOPS_BACKUP_ENCRYPTION_KEY / STELLAOPS_BACKUP_SIGNING_KEY env values, which the script materialises into private 0600 temp key files.

Restoring from an offline backup

# 1. Decrypt the opaque artifact back to the plaintext tar.gz (manifest verified first)
stellaops-backup-crypto decrypt \
  --in  /media/usb/stellaops/stellaops-backup-<ts>.tar.gz.stlebk \
  --out /tmp/stellaops-restore.tar.gz \
  --manifest /media/usb/stellaops/stellaops-backup-<ts>.tar.gz.stlebk.manifest.json \
  --kek-file /secure/backup-kek.key \
  --signing-key-file /secure/backup-sign.key

# 2. Restore the volumes from the decrypted tar.gz (wipe + extract + compose up), as in SP-003.

Monitoring

There is no dedicated Grafana “Backup Status” dashboard shipped in the repo. Backup health is surfaced through the stella doctor storage checks and the drill result JSON. To watch backup freshness, schedule stella doctor run --check check.storage.backup --format json and alert on a warn/fail severity, or check the latest artifacts/qa/backup-restore-drill/result-<ts>.json "status" field after each drill.

A Grafana backup dashboard is Draft/roadmap — wire one only after the backup metrics above are actually emitted by a service.

Evidence Capture

Generate a diagnostic bundle for support (the real command — there is no stella backup diagnostics):

stella doctor export -o /tmp/stellaops-diag-$(date +%Y%m%dT%H%M%S).zip

stella doctor export runs the doctor checks, packages the report plus recent logs and (unless --no-config) configuration into a ZIP, and prints a pass/warn/fail summary. Source: src/Cli/StellaOps.Cli/Commands/DoctorCommandGroup.cs.

The backup + restore drill also leaves auditable evidence under artifacts/qa/backup-restore-drill/: the pg-dump-<ts>.sql, the pre/post row-counts-*.txt, and the structured result-<ts>.json.

Escalation Path

1. L1 (On-call): Re-run failed backups (backup.sh --dry-run to diagnose), basic disk-space triage via stella doctor run --check check.storage.diskspace.
2. L2 (Platform / DevOps team): Restore drills, volume wipe + restore, schedule adjustments (host cron / systemd).
3. L3 (Architecture): Disaster-recovery execution and any PITR / WAL archiving design work (currently NOT IMPLEMENTED).

Last updated: 2026-05-31 (UTC) — reconciled against src/ and devops/; corrected the stella evidence command-group claims (DR-003) after verifying EvidenceCommandGroup.cs.