Evidence Migration Guide

Audience: Evidence Locker operators and platform DBAs performing upgrades, schema migrations, or disaster recovery. Purpose: Preserve the cryptographic chain-of-custody of sealed evidence across upgrades, schema changes, and recovery — using the commands and procedures that exist today, and clearly flagging the roadmap surface that does not.

This guide covers evidence-specific migration procedures during upgrades, schema changes, or disaster-recovery scenarios.

Status note (doc ⇄ code reconciliation). The Evidence Locker stores sealed evidence as an append-only, immutable chain: rows cannot be deleted, root_hash cannot be rewritten, and only the superseded_by link may transition (once, NULL → not-NULL). See 008_append_only_bundles.sql in StellaOps.EvidenceLocker.Infrastructure/Db/Migrations/. Consequences for this guide:

• There is no in-place mutation of sealed evidence. “Reindex” recomputes and verifies roots; it never overwrites them. A divergent recompute is reported as an error to investigate, not silently fixed.
• Schema migrations are forward-only (see ADR-004). The supported recovery from a bad migration is a PostgreSQL snapshot restore, not a logical row-level rollback — see the migration-recovery runbook.
• The stella evidence reindex, verify-continuity, and migrate CLI commands are wired but not yet serviceable end-to-end: they POST to /api/v1/evidence/reindex/*, /api/v1/evidence/continuity/*, and /api/v1/evidence/migrate/*, and those HTTP endpoints are not implemented in the Evidence Locker WebService. They will fail with a connection/404 error against a live service. Treat these flows as Draft / roadmap until the server endpoints land. The underlying reindex/continuity engine (IEvidenceReindexService) exists but is currently exercised only via DI and integration tests.

Overview

Evidence bundles are cryptographically linked data structures that must maintain integrity across upgrades. This guide ensures chain-of-custody is preserved during migrations.

Quick Reference

The CLI surface below maps to src/Cli/StellaOps.Cli/Commands/EvidenceCommandGroup.cs. Commands marked (roadmap) depend on Evidence Locker server endpoints that are not yet implemented (see status note above).

There is no stella evidence export --all. stella evidence export takes a single <bundle-id> argument and streams that bundle from POST /api/v1/bundles/{bundleId}/export. For a multi-type / date-ranged corpus export use stella airgap export-evidence (see Air-Gap Migration).

Pre-Migration Checklist

The Evidence Locker exposes per-bundle root_hash values (lowercase-hex SHA-256, enforced by the ^[0-9a-f]{64}$ column check in 008_append_only_bundles.sql), not a single global Merkle root. There is no stella evidence stats, roots-export, verify-all, or verify-bundle command — those were aspirational in earlier drafts and have no implementation. The grounded steps below use commands that exist today plus a PostgreSQL snapshot for the actual backup.

Schema/table names below are the real ones from source. Sealed bundles live in the evidence_locker.evidence_bundlestable (schema evidence_locker, created by 001_initial_schema.sql); there is no evidence schema and no evidence.bundlestable. The pg_dump/pg_restore examples therefore target --schema=evidence_locker. The Attestor verdict ledger (attestor.verdict_ledger, wired in the WebService Program.cs) is a separate attestor schema; include it only if your deployment co-locates it in the same database.

1. Capture Current State

# Snapshot the per-bundle root hashes that exist today.
# (Read the evidence_locker.evidence_bundles table directly; there is no roots-export CLI.)
# -At uses the default '|' field separator, matched by the cut -d'|' below.
psql -d stellaops -At \
  -c "SELECT bundle_id, root_hash, sealed_at FROM evidence_locker.evidence_bundles ORDER BY bundle_id" \
  > pre-migration-roots.txt

# Spot-check integrity of a known head bundle by re-exporting and verifying it.
# `verify` checks checksums + manifest + DSSE structure on an exported archive.
stella evidence export <bundle-id> --output /tmp/pre-check.tar.gz
stella evidence verify /tmp/pre-check.tar.gz --offline
if [ $? -ne 0 ]; then
  echo "ABORT: Evidence integrity check failed"
  exit 1
fi

2. Create Evidence Backup

Evidence is append-only and lives in PostgreSQL plus the object store, so the authoritative backup is a database snapshot (forward-only migration policy means recovery is by snapshot restore, not row rollback — see migration-recovery runbook).

# Snapshot the evidence_locker schema (and the object store, if file-system backed).
# Add --schema=attestor only if the attestor verdict ledger shares this database.
pg_dump -d stellaops --schema=evidence_locker --schema=attestor \
  -Fc -f /backup/evidence-$(date +%Y%m%d).dump

# Optionally export individual sealed bundles as portable archives for
# offline retention. `--include-rekor-proofs` and `--include-layers` are the
# real flags (there is no --include-attestations / --include-proofs).
stella evidence export <bundle-id> \
  --include-layers \
  --include-rekor-proofs \
  --output /backup/evidence-$(date +%Y%m%d)/<bundle-id>.tar.gz

# Verify each exported archive (checksums, manifest, DSSE envelope structure).
stella evidence verify /backup/evidence-*/<bundle-id>.tar.gz --offline

3. Document Chain-of-Custody

# Record the head bundle's per-bundle root hash from the snapshot above.
# Substitute <bundle-id> for the chain head you are tracking.
OLD_ROOT=$(grep '^<bundle-id>|' pre-migration-roots.txt | cut -d'|' -f2)
echo "Pre-migration root hash for <bundle-id>: ${OLD_ROOT}" > custody-log.txt
date >> custody-log.txt

Migration Procedures

Roadmap. The three procedures in this section drive the migrate, reindex, and verify-continuity CLI commands. Those commands are implemented in the CLI but call Evidence Locker server endpoints (/api/v1/evidence/migrate/*, /api/v1/evidence/reindex/*, /api/v1/evidence/continuity/*) that are not yet served — see the status note at the top. The exact CLI options below match EvidenceCommandGroup.cs so they are accurate once the server side lands.

Schema Migration (Version Upgrade)

When upgrading between versions with schema changes:

# Step 1: Assess migration impact (dry-run prints the migration plan).
# --from-version is required; --to-version defaults to "latest".
stella evidence migrate \
  --from-version 1.0 \
  --to-version 2.0 \
  --dry-run

# Step 2: Review migration plan output
# Ensure all steps are expected

# Step 3: Execute migration (prompts for confirmation)
stella evidence migrate \
  --from-version 1.0 \
  --to-version 2.0

The CLI migrate --rollback flag exists, but at the database layer migrations are forward-only (ADR-004). Recovery from a failed migration is a PostgreSQL snapshot restore — see Restore from Backup and the migration-recovery runbook.

Evidence Reindex (Continuity Recompute)

When the manifest hashing or Merkle tree construction changes, reindex recomputes and verifies each bundle’s root hash. Because sealed evidence is immutable, reindex does not rewrite the stored root_hash; a recompute that diverges from the sealed value is reported as an error for an operator to investigate, not silently corrected.

# Step 1: Assess reindex impact (dry-run). Optional --output writes the
# assessment JSON. Default --batch-size is 100.
stella evidence reindex \
  --dry-run \
  --output reindex-plan.json

# Review reindex-plan.json for:
# - Total records affected
# - Records to reindex
# - Estimated duration
# - Current -> target schema version

# Step 2: Execute reindex with batching (prompts for confirmation).
# --since filters to bundles created after the given ISO-8601 instant.
stella evidence reindex \
  --batch-size 100 \
  --since 2026-01-01

There is no --parallel option and no separate --progress subcommand; reindex streams progress inline. There is no roots-export command to capture a “new global root” — query evidence_locker.evidence_bundles.root_hash per bundle as in the pre-migration checklist if you need to record post-reindex roots.

Chain-of-Custody Verification

After a reindex, verify that a given before/after root pair is consistent. --old-root and --new-root are required; --format is one of json (default), html, or text.

# Verify continuity between the old and recomputed root for a bundle chain.
stella evidence verify-continuity \
  --old-root "${OLD_ROOT}" \
  --new-root "${NEW_ROOT}" \
  --output continuity-report.html \
  --format html

# Check verification results
if grep -q "FAIL" continuity-report.html; then
  echo "ERROR: Chain-of-custody verification failed!"
  echo "Review continuity-report.html for details"
  exit 1
fi

Rollback Procedures

Sealed evidence is append-only: rows are never deleted and root_hash is never rewritten (immutability trigger in 008_append_only_bundles.sql). There is no logical “undo” of sealed bundles — to record a corrected state you seal a new bundle and link it via the supersede chain. The only true rollback of a schema migration is a PostgreSQL snapshot restore.

Migration Rollback (Roadmap CLI)

# The migrate command exposes a --rollback flag that targets the server
# /api/v1/evidence/migrate/rollback endpoint (not yet implemented). It reuses
# --from-version to identify the migration to revert and prompts for confirmation.
stella evidence migrate \
  --rollback \
  --from-version 2.0

There is no stella evidence migrate --status(nor --logs / --resume). Until the server endpoints exist, do not rely on this command for recovery; use the snapshot restore below.

Restore from Backup

This is the authoritative recovery path (forward-only migration policy; see ADR-004 and the migration-recovery runbook).

# Step 1: Stop evidence-related services through the supported deployment path.
# Container name: stellaops-evidence-locker-web (compose service evidence-locker-web).
docker compose stop evidence-locker-web evidence-locker-worker

# Step 2: Restore the evidence_locker (and attestor) schema from the snapshot dump.
# Use --clean --if-exists so the restore replaces current objects cleanly.
# Match the schema list to what was captured by pg_dump above.
pg_restore -d stellaops --clean --if-exists \
  --schema=evidence_locker --schema=attestor \
  /backup/evidence-YYYYMMDD.dump

# Step 3: Restore any file-system object-store contents from the backup media,
# if the deployment uses the file-system object store (/data/evidence). There is
# no `stella evidence import` command; object-store files are restored out-of-band.

# Step 4: Spot-verify a restored head bundle by re-exporting and verifying it.
stella evidence export <bundle-id> --output /tmp/post-restore.tar.gz
stella evidence verify /tmp/post-restore.tar.gz --offline

# Step 5: Restart services
docker compose up -d evidence-locker-web evidence-locker-worker

Air-Gap Migration

For air-gapped environments without network access. The real air-gap export is stella airgap export-evidence (see src/Cli/StellaOps.Cli/Commands/CommandFactory.cs), distinct from the single-bundle stella evidence export.

Export Phase (Online Environment)

# Export a portable evidence package. --include accepts attestations, sboms,
# scans, vex, or all (default all). --from/--to bound the date range (ISO-8601);
# --tenant and --subject narrow scope; --verify checks signatures before export.
stella airgap export-evidence \
  --include all \
  --compress \
  --verify \
  --output /media/airgap-evidence/

# Generate checksums for the exported package
sha256sum /media/airgap-evidence/* > /media/checksums.txt

Transfer Phase

1. Copy to removable media
2. Verify checksums at destination
3. Scan media for security

Import Phase (Air-Gap Environment)

# Verify transfer integrity
sha256sum -c /media/checksums.txt

There is no stella evidence import command. Portable evidence is consumed in the air-gapped environment either by importing the air-gap bundle through the air-gap import tooling, or by restoring the evidence schema from a PostgreSQL snapshot (see Restore from Backup). Individual sealed bundles can be re-downloaded as self-verifying portable archives from GET /evidence/{bundleId:guid}/portable (the route requires the bundle UUID and the evidence:read scope) and validated offline:

# Validate a portable bundle archive offline (no Rekor/network access).
stella evidence verify /media/airgap-evidence/<bundle-id>.tar.gz --offline

Troubleshooting

The CLI does not provide migrate --status/--logs/--resume, integrity-check, stats, orphans, or reconcile commands. The guidance below uses the commands and queries that exist today.

Migration Stuck or Timeout

The migrate command runs synchronously against the (roadmap) server endpoint; there is no resume/checkpoint CLI. If a migration attempt does not complete:

1. Inspect Evidence Locker service logs (docker compose logs evidence-locker-web).
2. Recover by restoring the pre-migration PostgreSQL snapshot — see Restore from Backup and the migration-recovery runbook.

Root Hash Mismatch

If a reindex reports a recomputed root_hash that differs from the sealed value:

1. Do not proceed with the upgrade — a mismatch means the manifest was tampered with or the referenced materials no longer hash to the recorded values. Sealed evidence is immutable, so reindex will not “fix” it.
2. Cross-reference the bundle through its supersede chain. The /chain endpoint is route-constrained to a GUID (/evidence/{bundleId:guid}/chain), so <bundle-id> here must be the bundle’s UUID bundle_id, not the eb-... display id from stella evidence export:

   # Walk the supersede chain (oldest-first) for the affected bundle.
   # Requires the evidence:read scope on $TOKEN.
   curl -H "Authorization: Bearer $TOKEN" \
     "$EVIDENCE_URL/evidence/<bundle-uuid>/chain"
   

3. Review recent changes to the evidence store and object backend.
4. Escalate with the chain output and the divergent recomputed hash.

Missing Evidence Records

There is no stats/orphans/reconcile CLI. Inspect the store directly:

# Count sealed bundles by kind (kind is a smallint, 1-3, per 008_append_only_bundles.sql).
psql -d stellaops -c \
  "SELECT kind, count(*) FROM evidence_locker.evidence_bundles GROUP BY kind ORDER BY kind"

# List current chain heads (superseded_by IS NULL) with their storage key and chain links.
psql -d stellaops -c \
  "SELECT bundle_id, storage_key, supersedes, superseded_by FROM evidence_locker.evidence_bundles \
   WHERE superseded_by IS NULL ORDER BY sealed_at DESC LIMIT 50"

The Evidence Locker worker (stellaops-evidence-locker-worker) does not currently run a background integrity-verification loop. Its ExecuteAsync (StellaOps.EvidenceLocker.Worker/Worker.cs) only verifies database connectivity once at startup, logs the result, then idles (Task.Delay(Timeout.Infinite)). Treat its logs as a liveness/connectivity signal, not a record-reconciliation report; use the direct SQL above to inspect the store.

Performance Issues

For large evidence stores, tune the reindex batch size (default 100). There is no --parallel option and no --progress subcommand — progress streams inline during execution.

# Larger batches reduce round-trips; --since narrows the working set.
stella evidence reindex \
  --batch-size 500 \
  --since 2026-01-01

Audit Trail Requirements

All evidence migrations must maintain audit trail:

Sealed root_hash values do not change in place — a corrected state is a new sealed bundle linked via the supersede chain, so the audit record captures the predecessor/successor pair rather than a mutated root.

Related Documents

• Upgrade Runbook - Overall upgrade procedures
• Blue-Green Deployment - Zero-downtime deployment
• Evidence Locker Architecture - Technical design
• Migration Recovery Runbook - Forward-only migration recovery via snapshot restore
• ADR-004: Forward-Only Migrations - Migration/rollback policy
• Air-Gap Operations - Offline deployment guide