Authority Backup & Restore Runbook

Audience: Authority operators and SREs responsible for protecting identity data and signing keys.

Purpose: Provide step-by-step hot/cold backup, restore, and disaster-recovery procedures for a Stella Ops Authority deployment.

Scope

Inventory Checklist

ComponentLocation (compose default)Notes
PostgreSQL datapostgres-data volume (/var/lib/docker/volumes/.../postgres-data)Contains all Authority tables (authority_user, authority_client, authority_token, etc.).
Configurationetc/authority.yamlMounted read-only into the container at /etc/authority.yaml.
Plugin manifestsetc/authority.plugins/*.yamlIncludes standard.yaml with tokenSigning.keyDirectory.
Signing keysauthority-keys volume -> /app/keysPath is derived from tokenSigning.keyDirectory (defaults to ../keys relative to the manifest).

TIP: Confirm the deployed key directory via tokenSigning.keyDirectory in etc/authority.plugins/standard.yaml; some installations relocate keys to /var/lib/stellaops/authority/keys.

Hot Backup (no downtime)

  1. Create output directory: mkdir -p backup/$(date +%Y-%m-%d) on the host.
  2. Dump PostgreSQL:
    docker compose -f ops/authority/docker-compose.authority.yaml exec postgres \
  pg_dump -Fc -d stellaops-authority \
  -f /dump/authority-$(date +%Y%m%dT%H%M%SZ).dump
docker compose -f ops/authority/docker-compose.authority.yaml cp \
  postgres:/dump/authority-$(date +%Y%m%dT%H%M%SZ).dump backup/
    The pg_dump archive preserves indexes and can be restored with pg_restore.
  3. Capture configuration + manifests:
    cp devops/etc/authority.yaml backup/
rsync -a devops/etc/authority.plugins/ backup/authority.plugins/
  4. Export signing keys: the compose file maps authority-keys to a local Docker volume. Snapshot it without stopping the service:
    docker run --rm \
  -v authority-keys:/keys \
  -v "$(pwd)/backup:/backup" \
  busybox tar czf /backup/authority-keys-$(date +%Y%m%dT%H%M%SZ).tar.gz -C /keys .
  5. Checksum: generate SHA-256 digests for every file and store them alongside the artefacts.
  6. Encrypt & upload: wrap the backup folder using your secrets management standard (e.g., age, GPG) and upload to the designated offline vault.

Cold Backup (planned downtime)

  1. Notify stakeholders and drain traffic (CLI clients should refresh tokens afterwards).
  2. Stop services:
    docker compose -f ops/authority/docker-compose.authority.yaml down
  3. Back up volumes directly using tar:
    docker run --rm -v postgres-data:/data -v "$(pwd)/backup:/backup" \
  busybox tar czf /backup/postgres-data-$(date +%Y%m%d).tar.gz -C /data .
docker run --rm -v authority-keys:/keys -v "$(pwd)/backup:/backup" \
  busybox tar czf /backup/authority-keys-$(date +%Y%m%d).tar.gz -C /keys .
  4. Copy configuration + manifests as in the hot backup (steps 3–6).
  5. Restart services and verify health:
    docker compose -f ops/authority/docker-compose.authority.yaml up -d
curl -fsS http://localhost:8080/ready

Restore Procedure

  1. Provision clean volumes: remove existing volumes if you’re rebuilding a node (docker volume rm postgres-data authority-keys), then recreate the compose stack so empty volumes exist.
  2. Restore PostgreSQL:
    docker compose exec -T postgres pg_restore -d stellaops-authority --clean < backup/authority-YYYYMMDDTHHMMSSZ.dump
    Use --clean to drop existing objects before restoring; omit if doing a partial restore.
  3. Restore configuration/manifests: copy authority.yaml and authority.plugins/* into place before starting the Authority container.
  4. Restore signing keys: untar into the mounted volume:
    docker run --rm -v authority-keys:/keys -v "$(pwd)/backup:/backup" \
  busybox tar xzf /backup/authority-keys-YYYYMMDD.tar.gz -C /keys
    Ensure file permissions remain 600 for private keys (chmod -R 600).
  5. Start services & validate:
    docker compose up -d
curl -fsS http://localhost:8080/health
  6. Validate JWKS and tokens: call /jwks and issue a short-lived token via the CLI to confirm key material matches expectations. If the restored environment requires a fresh signing key, follow the rotation SOP in AUTHORITY.mdusing ops/authority/key-rotation.sh to invoke /internal/signing/rotate.

Disaster Recovery Notes

Verification Checklist

See Also