Evidence Migration Guide

Currency (2026-10-07): current source composes evidence-web and evidence-worker over one family database. The export target is /api/evidence/v1/bundles/{bundleId}/export; verify the deployed build before an operational window.

Audience: Evidence Locker operators and platform DBAs performing upgrades, schema migrations, or disaster recovery. Purpose: Preserve the cryptographic chain-of-custody of sealed evidence across upgrades, schema changes, and recovery — using only the commands and procedures that exist today.

This guide covers evidence-specific migration procedures during upgrades, schema changes, or disaster-recovery scenarios.

Status note (doc ⇄ code reconciliation). The Evidence Locker stores sealed evidence as an append-only, immutable chain: rows cannot be deleted, root_hash cannot be rewritten, and only the superseded_by link may transition (once, NULL → not-NULL). See 008_append_only_bundles.sql in StellaOps.EvidenceLocker.Infrastructure/Db/Migrations/. Consequences for this guide:

  • There is no in-place mutation of sealed evidence. “Reindex” recomputes and verifies roots; it never overwrites them. A divergent recompute is reported as an error to investigate, not silently fixed.
  • Schema migrations are forward-only (see ADR-004). The supported recovery from a bad migration is a PostgreSQL snapshot restore, not a logical row-level rollback — see the migration-recovery runbook.
  • The evidence reindex, verify-continuity and migrate CLI commands were deleted on 2026-09-03 (SPRINT_20260903_005, EVD-DEL-2). They posted to /api/v1/evidence/reindex/*, /api/v1/evidence/continuity/verify and /api/v1/evidence/migrate/*, and no host has ever mapped those routes — neither the Evidence Locker WebService nor the consolidated Attestor host — so the commands could only ever fail. That CLI surface is gone, not pending. The reindex/continuity engine itself is unaffected and still ships as a library (IEvidenceReindexService, exercised through DI and integration tests); a CLI surface returns only when an owner route exists to drive it.

Overview

Evidence bundles are cryptographically linked data structures that must maintain integrity across upgrades. This guide ensures chain-of-custody is preserved during migrations.

Quick Reference

The CLI surface below maps to src/Cli/StellaOps.Cli/Commands/EvidenceCommandGroup.cs. The guide no longer lists a roadmap CLI surface: rows whose command was deleted now name what an operator actually does instead (see the status note above).

ScenarioCLI CommandRisk LevelDowntime
Schema migrationPostgreSQL snapshot restore (no CLI command)MediumMinutes
Reindex / continuity recomputenone shipped — IEvidenceReindexService is driven in-processLowNone
Single evidence bundle exportstella evidence export <bundle-id>NoneNone
Air-gap evidence export (all types)stella airgap export-evidence --include allNoneNone

There is no stella evidence export --all. stella evidence export takes a single <bundle-id> argument and initiates the asynchronous job at POST /api/evidence/v1/bundles/{bundleId}/export. For a multi-type / date-ranged corpus export use stella airgap export-evidence (see Air-Gap Migration).

Pre-Migration Checklist

The Evidence Locker exposes per-bundle root_hash values (lowercase-hex SHA-256, enforced by the ^[0-9a-f]{64}$ column check in 008_append_only_bundles.sql), not a single global Merkle root. There is no stella evidence stats, roots-export, verify-all, or verify-bundle command — those were aspirational in earlier drafts and have no implementation. The grounded steps below use commands that exist today plus a PostgreSQL snapshot for the actual backup.

Current ownership (source-verified 2026-10-07). The Evidence family owns the stellaops_evidence database. Sealed bundles remain in evidence_locker.evidence_bundles; the family also owns attestor, proofchain, and evidence schemas. Capture the whole family database and its object store so migration state, legal holds, proof chains, and timestamp-assurance records remain together. The startup ledger is evidence.schema_migrations; there is no evidence.bundles table.

1. Capture Current State

# Snapshot the per-bundle root hashes that exist today.
# (Read the evidence_locker.evidence_bundles table directly; there is no roots-export CLI.)
# -At uses the default '|' field separator, matched by the cut -d'|' below.
psql -d stellaops_evidence -At \
  -c "SELECT bundle_id, root_hash, sealed_at FROM evidence_locker.evidence_bundles ORDER BY bundle_id" \
  > pre-migration-roots.txt

# Spot-check integrity of a known head bundle by re-exporting and verifying it.
# `verify` checks checksums + manifest + DSSE structure on an exported archive.
stella evidence export <bundle-id> --output /tmp/pre-check.tar.gz
stella evidence verify /tmp/pre-check.tar.gz --offline
if [ $? -ne 0 ]; then
  echo "ABORT: Evidence integrity check failed"
  exit 1
fi

2. Create Evidence Backup

Evidence is append-only and lives in PostgreSQL plus the object store, so the authoritative backup is a database snapshot (forward-only migration policy means recovery is by snapshot restore, not row rollback — see migration-recovery runbook).

# Snapshot the entire Evidence family database and the configured object store.
pg_dump -d stellaops_evidence \
  -Fc -f /backup/evidence-$(date +%Y%m%d).dump

# Optionally export individual sealed bundles as portable archives for
# offline retention. `--include-rekor-proofs` and `--include-layers` are the
# real flags (there is no --include-attestations / --include-proofs).
stella evidence export <bundle-id> \
  --include-layers \
  --include-rekor-proofs \
  --output /backup/evidence-$(date +%Y%m%d)/<bundle-id>.tar.gz

# Verify each exported archive (checksums, manifest, DSSE envelope structure).
stella evidence verify /backup/evidence-*/<bundle-id>.tar.gz --offline

3. Document Chain-of-Custody

# Record the head bundle's per-bundle root hash from the snapshot above.
# Substitute <bundle-id> for the chain head you are tracking.
OLD_ROOT=$(grep '^<bundle-id>|' pre-migration-roots.txt | cut -d'|' -f2)
echo "Pre-migration root hash for <bundle-id>: ${OLD_ROOT}" > custody-log.txt
date >> custody-log.txt

Migration Procedures

Schema Migration (Version Upgrade)

Both Evidence roles converge the family schema at startup through AddConsolidatedEvidencePersistence and AddStartupMigrations in src/Evidence/__Libraries/StellaOps.Evidence.Persistence/Extensions/ConsolidatedEvidencePersistenceExtensions.cs. The service-specific resolver requires STELLAOPS_POSTGRES_EVIDENCE_CONNECTION (or Postgres:Evidence:ConnectionString); no predecessor migration switch selects a second authority. Migrations remain forward-only under ADR-004. There is no operator CLI that plans, applies, or reverts the Evidence schema migration.

The operator’s job around a version upgrade is therefore the checklist above (capture roots, snapshot the database) plus verification afterwards — not driving the migration. If a migration lands badly, the supported recovery is a PostgreSQL snapshot restore, see Restore from Backup and the migration-recovery runbook.

Evidence Reindex (Continuity Recompute)

When manifest hashing or Merkle tree construction changes, a reindex recomputes and verifies each bundle’s root hash. Because sealed evidence is immutable, a reindex does not rewrite the stored root_hash; a recompute that diverges from the sealed value is an error for an operator to investigate, not something to correct in place.

That engine is IEvidenceReindexService (StellaOps.EvidenceLocker.Core/Reindexing/), and it is reachable only in-process — it is registered in the Evidence Locker’s DI container and covered by EvidenceReindexServiceTests / EvidenceReindexIntegrationTests. No server route and no CLI command expose it. Until an owner route exists, a recompute is an engineering task against the service, not an operator procedure. To record roots before and after such a task, query evidence_locker.evidence_bundles.root_hash per bundle as in the pre-migration checklist.

Rollback Procedures

Sealed evidence is append-only: rows are never deleted and root_hash is never rewritten (immutability trigger in 008_append_only_bundles.sql). There is no logical “undo” of sealed bundles — to record a corrected state you seal a new bundle and link it via the supersede chain. The only true rollback of a schema migration is a PostgreSQL snapshot restore.

Migration Rollback

There is no evidence migration rollback command. The CLI migrate --rollback flag was deleted with the rest of the migrate command on 2026-09-03; it targeted /api/v1/evidence/migrate/rollback, which no host serves. Database migrations are forward-only (ADR-004) regardless, so the snapshot restore below is the recovery path.

Restore from Backup

This is the authoritative recovery path (forward-only migration policy; see ADR-004 and the migration-recovery runbook).

# Step 1: Stop evidence-related services through the supported deployment path.
# Container name: stellaops-evidence-web (compose service evidence-web).
docker compose stop evidence-web evidence-worker

# Step 2: Restore the complete Evidence family database from its matching snapshot.
# Use --clean --if-exists so the restore replaces current objects cleanly.
# Match the schema list to what was captured by pg_dump above.
pg_restore -d stellaops_evidence --clean --if-exists \
  /backup/evidence-YYYYMMDD.dump

# Step 3: Restore any file-system object-store contents from the backup media,
# if the deployment uses the file-system object store (/data/evidence). There is
# no `stella evidence import` command; object-store files are restored out-of-band.

# Step 4: Restart the Evidence roles through the approved Compose chain.
docker compose up -d evidence-web evidence-worker

# Step 5: After readiness, re-export and verify a restored head bundle.
stella evidence export <bundle-id> --output /tmp/post-restore.tar.gz
stella evidence verify /tmp/post-restore.tar.gz --offline

Air-Gap Migration

For air-gapped environments without network access. The real air-gap export is stella airgap export-evidence (see src/Cli/StellaOps.Cli/Commands/CommandFactory.cs), distinct from the single-bundle stella evidence export.

Export Phase (Online Environment)

# Export a portable evidence package. --include accepts attestations, sboms,
# scans, vex, or all (default all). --from/--to bound the date range (ISO-8601);
# --tenant and --subject narrow scope; --verify checks signatures before export.
stella airgap export-evidence \
  --include all \
  --compress \
  --verify \
  --output /media/airgap-evidence/

# Generate checksums for the exported package
sha256sum /media/airgap-evidence/* > /media/checksums.txt

Transfer Phase

  1. Copy to removable media
  2. Verify checksums at destination
  3. Scan media for security

Import Phase (Air-Gap Environment)

# Verify transfer integrity
sha256sum -c /media/checksums.txt

There is no stella evidence import command. Portable evidence is consumed in the air-gapped environment either by importing the air-gap bundle through the air-gap import tooling, or by restoring the evidence schema from a PostgreSQL snapshot (see Restore from Backup). Individual sealed bundles can be re-downloaded as self-verifying portable archives from GET /api/evidence/v1/evidence/{bundleId:guid}/portable (the route requires the bundle UUID and the evidence:read scope) and validated offline:

# Validate a portable bundle archive offline (no Rekor/network access).
stella evidence verify /media/airgap-evidence/<bundle-id>.tar.gz --offline

Troubleshooting

The CLI provides no evidence migrate, reindex, verify-continuity, integrity-check, stats, orphans, or reconcile command. The guidance below uses the commands and queries that exist today.

Migration Stuck or Timeout

Schema migration happens inside the service at startup, so a stuck migration is a service-startup problem, not a command to resume:

  1. Inspect Evidence service logs (docker compose logs evidence-web) — EVD-10a retired the evidence-locker-web host into the consolidated Evidence service.
  2. Recover by restoring the pre-migration PostgreSQL snapshot — see Restore from Backup and the migration-recovery runbook.

Root Hash Mismatch

If a reindex reports a recomputed root_hash that differs from the sealed value:

  1. Do not proceed with the upgrade — a mismatch means the manifest was tampered with or the referenced materials no longer hash to the recorded values. Sealed evidence is immutable, so reindex will not “fix” it.
  2. Cross-reference the bundle through its supersede chain. The /chain endpoint is route-constrained to a GUID (/api/evidence/v1/evidence/{bundleId:guid}/chain), so <bundle-id> here must be the bundle’s UUID bundle_id, not the eb-... display id from stella evidence export:
    # Walk the supersede chain (oldest-first) for the affected bundle.
# Requires the evidence:read scope on $TOKEN.
curl -H "Authorization: Bearer $TOKEN" \
  "$EVIDENCE_URL/api/evidence/v1/evidence/<bundle-uuid>/chain"
  3. Review recent changes to the evidence store and object backend.
  4. Escalate with the chain output and the divergent recomputed hash.

Missing Evidence Records

There is no stats/orphans/reconcile CLI. Inspect the store directly:

# Count sealed bundles by kind (kind is a smallint, 1-3, per 008_append_only_bundles.sql).
psql -d stellaops_evidence -c \
  "SELECT kind, count(*) FROM evidence_locker.evidence_bundles GROUP BY kind ORDER BY kind"

# List current chain heads (superseded_by IS NULL) with their storage key and chain links.
psql -d stellaops_evidence -c \
  "SELECT bundle_id, storage_key, supersedes, superseded_by FROM evidence_locker.evidence_bundles \
   WHERE superseded_by IS NULL ORDER BY sealed_at DESC LIMIT 50"

Current worker composition (source-verified 2026-10-07): evidence-worker registers attestation retry/bulk workers, optional external-sync work, capsule retention, legacy PII audit, and Timestamp Assurance schedules. See src/Evidence/StellaOps.Evidence.Worker/EvidenceWorkerComposition.cs. These loops do not constitute a whole-store bundle-integrity reconciliation report. Use the direct reads above for the stated store questions and retain their tenant scope.

Performance Issues

For large evidence stores, batching and date-window filtering are properties of the in-process reindex engine (IEvidenceReindexService), not of an operator command — there is no CLI verb to tune. Size the work at the call site that drives the service.

Audit Trail Requirements

All evidence migrations must maintain audit trail:

EventRequired DataRetention
Migration StartTimestamp, version, operatorPermanent
Schema ChangeBefore/after schema versionsPermanent
Bundle SupersedePredecessor bundle id, successor bundle id, both root hashesPermanent
Verification / Continuity CheckPass/fail, anomalies, timestamps7 years
Snapshot RestoreReason, snapshot timestamp, restored schema(s)Permanent

Sealed root_hash values do not change in place — a corrected state is a new sealed bundle linked via the supersede chain, so the audit record captures the predecessor/successor pair rather than a mutated root.