Evidence Locker Operations Runbook

Audience: Platform and on-call engineers operating the Stella Ops Evidence Locker — the tamper-evident store for release evidence bundles, decision capsules, verdict attestations, and the regulatory artifact ledger. Use this runbook to verify bundles, manage retention and legal holds, and recover the locker after a failure.

Status: RECONCILED AGAINST SOURCE (2026-05-31 UTC)

Reconciliation note (2026-05-31): This runbook was reconciled against src/EvidenceLocker, the CLI evidence command groups (src/Cli/StellaOps.Cli/Commands/EvidenceCommandGroup.cs, EvidenceHoldsCommandGroup.cs), the Doctor Evidence Locker plugin (src/Doctor/__Plugins/StellaOps.Doctor.Plugin.EvidenceLocker), the Doctor engine category filter (src/__Libraries/StellaOps.Doctor/Engine/CheckRegistry.cs), the WebService route map (src/EvidenceLocker/.../StellaOps.EvidenceLocker.WebService/Program.cs + the Api/ endpoint files), and the canonical scope catalog (src/Authority/.../StellaOpsScopes.cs). Many commands in the earlier draft (stella evidence index|anchor|storage|cleanup|archive|verify-report| inspect|recover|mark-corrupted|exists|restore|diagnostics, stella backup, stella config set, stella service restart) and the four stella_evidence_* / stella_merkle_anchor_age_seconds metrics do not exist in the codebase and have been corrected, removed, or flagged NOT IMPLEMENTED below.

A second reconciliation pass (2026-05-31) found additional drift the first pass missed and corrected it inline: (a) stella doctor --category "Evidence Locker" does not work — the category filter only matches the DoctorCategory enum (Enum.TryParse), and the Evidence Locker plugin’s category is Security, while “Evidence Locker” is only its display name; use --check <id> or --category Security; (b) there is no stella logs <service> command — the platform log reader is stella obs logs --from <iso> --to <iso> --service evidence-locker (no positional service, no --since); © the reindex / verify-continuity / migrate CLI verbs target /api/v1/evidence/{reindex,continuity,migrate}/... endpoints that are not implemented server-side (the reindex service exists as a library only, IEvidenceReindexService, never HTTP-mapped); (d) the live evidence holds CLI is the simpler one wired in EvidenceCommandGroup (list, create --reason, release <hold-id>) — the richer EvidenceHoldsCommandGroup (--name/--scope/--digest, show, --confirm) is present in the tree but not wired into the command factory. The code is the source of truth; this doc has been brought in line with it.

Scope

Evidence locker operations: evidence bundle export/verification, decision-capsule lifecycle, verdict attestation storage, regulatory artifact ledger, retention sweeping, evidence re-indexing, and disaster recovery. The service is HTTP/CLI-driven; there is no bespoke “anchor chain” or “index maintenance” subsystem (see flagged items).

Pre-flight Checklist

Environment Verification

# Run the Evidence Locker health checks (Doctor plugin). Prefer running the
# individual checks by id — the plugin's Doctor *category* is `Security`
# (its display name is "Evidence Locker"), so a category filter must use the
# enum value, not the display name (see CORRECTION below).
stella doctor --check check.evidencelocker.retrieval    # Attestation Retrieval
stella doctor --check check.evidencelocker.index        # Evidence Index Consistency
stella doctor --check check.evidencelocker.provenance   # Provenance Chain Integrity
stella doctor --check check.evidencelocker.merkle       # Merkle Anchor Verification (only runs if EvidenceLocker:Anchoring:Enabled=true)

# Category filter (runs ALL Security-category checks, not just Evidence Locker):
stella doctor --category Security

# Liveness / readiness probes exposed by the WebService (no auth required):
curl -fsS http://<evidence-locker-host>/healthz       # liveness
curl -fsS http://<evidence-locker-host>/readyz        # readiness
curl -fsS http://<evidence-locker-host>/health/ready  # legacy readiness alias

CORRECTION (2026-05-31): stella doctor --category "Evidence Locker" does not work. The CLI passes --category to DoctorRunOptions.Categories, which is matched with Enum.TryParse<DoctorCategory>(...) in CheckRegistry.GetFilteredPlugins — only enum names (Core, Database, Security, Cryptography, Attestation, Docker, …) match. The Evidence Locker plugin declares Category => DoctorCategory.Security and DisplayName => "Evidence Locker"; passing the display name parses to no category, so zero checks run. Use the per-check --check check.evidencelocker.* ids (recommended — they target exactly this module) or --category Security (broader; runs every Security-category plugin).

NOTE: There is no stella evidence status, stella evidence index status, or stella evidence anchor verify command. Health and index/anchor consistency are surfaced through the Doctor checks above, not through dedicated evidence subcommands.

Metrics to Watch

FLAG — NOT IMPLEMENTED: The earlier draft listed stella_evidence_artifacts_total, stella_evidence_retrieval_latency_seconds, stella_evidence_storage_bytes, and stella_merkle_anchor_age_seconds. None of these metric names exist anywhere in src/ or devops/, and there is no “Evidence Locker” Grafana dashboard committed in the repo. Until the service emits a documented meter, monitor Evidence Locker health via:

  • the Doctor checks listed above (run on a schedule via the Doctor scheduler), and
  • the standard ASP.NET / /readyz health probes and host-level disk/storage metrics.

When dedicated metrics land, replace this block with the real meter and instrument names (and link the dashboard JSON under devops/).

Standard Procedures

SP-001: Evidence Bundle Verification

Frequency: On-demand (e.g. before an audit handoff), or after import on a downstream host Duration: Seconds to minutes depending on bundle size

The CLI verifies an exported bundle archive (it does not run a server-side “integrity sweep”).

  1. Export the bundle you want to check (or use one already exported):

    stella evidence export <bundle-id> --output bundle.tar.gz
# e.g. bundle-id = eb-2026-01-06-abc123

  2. Verify the exported archive (checksums, manifest, DSSE envelope structure, Rekor proofs):

    # Full verification
stella evidence verify bundle.tar.gz

# Air-gapped: skip Rekor transparency-log lookups
stella evidence verify bundle.tar.gz --offline

# Checksums + manifest only (skip DSSE signature structure check)
stella evidence verify bundle.tar.gz --skip-signatures

# Machine-readable output
stella evidence verify bundle.tar.gz --output json

    Exit code is 0 when all checks pass, non-zero otherwise.

  3. For a single-file evidence card (compact, signed) instead of a tarball:

    stella evidence card export <pack-id> --output pack.evidence-card.json
stella evidence card verify pack.evidence-card.json [--offline] [--trust-root <bundle>]

NOTE: stella evidence verify --mode quick|full and stella evidence verify-report do not exist. verify takes a bundle path and emits a pass/fail table (or --output json).

SP-002: Evidence Re-Index

Frequency: After a schema or hashing-algorithm change to evidence bundles Duration: Varies with record count (the assessment estimates duration)

Re-indexing recomputes Merkle roots for stored bundles. Always dry-run first.

  1. Assess impact (dry run):

    stella evidence reindex --dry-run [--since 2026-01-01T00:00:00Z] [--batch-size 100] \
  --output reindex-plan.json

  2. Execute (prompts for confirmation):

    stella evidence reindex [--since <iso8601>] [--batch-size 100]

  3. Verify chain-of-custody continuity across the old and new Merkle roots:

    stella evidence verify-continuity --old-root sha256:<old> --new-root sha256:<new> \
  --format json --output continuity-report.json

FLAG — NOT IMPLEMENTED (server side): The reindex and verify-continuity CLI verbs target /api/v1/evidence/reindex/{assess,execute} and /api/v1/evidence/continuity/verify, but those HTTP endpoints do not exist in the WebService — they appear only as client-side URL strings in EvidenceCommandGroup.cs and there are no Map* registrations for them anywhere in src/. A reindex engine exists as a library (IEvidenceReindexService / EvidenceReindexService, with ReindexAsync / VerifyContinuityAsync / GenerateCrossReferenceAsync / CreateCheckpointAsync / RollbackToCheckpointAsync), but it is not wired to any route. Running these CLI commands against a live service will fail with a connection error or 404. Until the endpoints are mapped, treat reindexing as a forward-looking / internal-only capability: drive it from the library in-process or via a maintenance task, not the CLI. Re-index is rarely needed in steady state (the DB auto-migrates forward — see SP-003).

SP-003: Schema Migration

Frequency: During upgrades that change the evidence schema version Duration: Per migration plan

# Show the migration plan without applying
stella evidence migrate --from-version <ver> [--to-version <ver>] --dry-run

# Apply (prompts for confirmation)
stella evidence migrate --from-version <ver> [--to-version <ver>]

# Roll back a previously failed migration
stella evidence migrate --from-version <ver> --rollback

FLAG — NOT IMPLEMENTED (server side): As with reindex/verify-continuity (SP-002), the migrate CLI verb posts to /api/v1/evidence/migrate/{plan,execute,rollback}, and those endpoints are not mapped in the WebService. The CLI command exists but has no backing route; it will fail against a live service. Do not rely on stella evidence migrate for production schema changes today.

The authoritative migration path is automatic: the service-owned PostgreSQL schema auto-migrates on startup (EvidenceLockerMigrationHostedService, migrations 001–012 under StellaOps.EvidenceLocker.Infrastructure/Db/Migrations — verified: 001_initial_schema … 012_capsule_retention_sweep) when EvidenceLocker:Database:ApplyMigrationsAtStartup is true (the default). Platform migration policy is forward-only (ADR-004); recover DDL state via PostgreSQL snapshot restore — see migration-recovery runbook. There is no implemented evidence-schema rollback verb; the --rollback flag posts to the unmapped /api/v1/evidence/migrate/rollback endpoint.

Frequency: Background (configurable); holds managed on-demand Duration: Varies

Retention is enforced by a background sweep, not a stella evidence cleanup command.

CORRECTION (2026-05-31): The earlier draft documented holds list [--status], holds create --name --scope --digest --component --from --to, holds show [--artifacts], and holds release --confirm --reason. Those flags belong to a SECOND, richer implementation (EvidenceHoldsCommandGroup) that exists in the source tree but is not wired into the command factory — CommandFactory.cs builds EvidenceCommandGroup.BuildEvidenceCommand, which adds the local simple BuildHoldsCommand (only list, create --reason, release <hold-id>; no show subcommand). Script against the minimal surface above until the richer group is wired in.

FLAG — Draft/roadmap: even the wired holds CLI currently renders sample/illustrative output and is not connected to a live holds API in the WebService. Treat hold IDs and counts in its output as placeholders. The real server-side legal-hold creation endpoint is POST /evidence/hold/{caseId} (requires evidence:create; see API reference) and storage-layer hold enforcement is real for S3 backends via Object Lock legal hold (see SP-005).

SP-005: Object Store & WORM / Object Lock

Frequency: Configuration-time; review during audits Duration: N/A

The evidence object store backend is selected by EvidenceLocker:ObjectStore:Kind. Kind is [Required] (enum FileSystem | AmazonS3) — there is no code-level default; the on-prem deployment config sets FileSystem:

Per repo policy, on-prem FileSystem is the default; do not switch to a cloud-managed backend as a default. To inspect effective configuration:

stella config list
stella config show EvidenceLocker

NOTE: stella config set does not exist — the CLI config group is read-only (list, show). Change configuration through the service’s config sources (env / files / DB overrides) and restart the service.

Incident Procedures

INC-001: Bundle Verification Failure

Symptoms:

Investigation:

# Re-run with JSON to capture which check failed
stella evidence verify bundle.tar.gz --output json > /tmp/verify.json

# Inspect the bundle / proof chain via the API (requires evidence:read)
curl -fsS -H "Authorization: Bearer $TOKEN" \
  http://<evidence-locker-host>/evidence/<bundleId>            # bundle metadata
curl -fsS -H "Authorization: Bearer $TOKEN" \
  http://<evidence-locker-host>/evidence/<bundleId>/chain      # custody / proof chain

Resolution:

  1. Transport corruption (re-export fixes it): re-run stella evidence export <bundle-id> and re-verify. The on-disk bundle in the locker is the source of truth.

  2. Checksum/manifest mismatch in the stored bundle: run the Doctor checks (check.evidencelocker.index, check.evidencelocker.provenance) to confirm whether the stored artifact or only the export is affected, then restore from backup (DR-001).

  3. Suspected false positive (tooling bug): verify with --skip-signatures to isolate the failing layer, capture the JSON output, and escalate to L2/L3.

INC-002: Evidence Retrieval Failure

Symptoms:

Investigation:

# Retrieval health check (probes the locker's ability to read attestation artifacts)
stella doctor --check check.evidencelocker.retrieval

# Index consistency between the index and stored artifacts
stella doctor --check check.evidencelocker.index

# Direct API probes (evidence:read)
curl -fsS -H "Authorization: Bearer $TOKEN" \
  http://<evidence-locker-host>/api/v1/evidence/packs
curl -fsS -H "Authorization: Bearer $TOKEN" \
  http://<evidence-locker-host>/api/v1/evidence/packs/<id>

Resolution:

  1. Object store unreachable: verify the configured EvidenceLocker:ObjectStore backend (filesystem mount present and writable, or S3 endpoint/credentials reachable). Check the service logs (stella obs logs --service evidence-locker --from <iso8601> --to <iso8601>) and /readyz. (CORRECTION: there is no stella logs <service> command; the platform log reader lives under obs and requires --from/--to rather than --since — see the note in the Evidence Capture section.)

  2. Index inconsistency: if check.evidencelocker.index flags drift, run a re-index (SP-002) after taking a backup. There is no stella evidence index rebuild command.

  3. Database / migration issue: confirm the schema converged on startup (migrations 001–012); a missing table is the most common cause of 5xx (see AGENTS.md §2.7).

INC-003: Merkle Anchor / Continuity Concern

Symptoms:

Investigation:

# Merkle/anchor health (no-op/skip unless anchoring is enabled)
stella doctor --check check.evidencelocker.merkle

# Verify continuity across roots after a reindex/upgrade
# (NOTE: this CLI verb posts to the unmapped /api/v1/evidence/continuity/verify endpoint —
#  see SP-002 NOT IMPLEMENTED flag — and will fail against a live service today.)
stella evidence verify-continuity --old-root sha256:<old> --new-root sha256:<new> --output table

Resolution:

  1. Continuity failure after reindex: treat the new root as unverified. Restore the pre-reindex state from backup (DR-001) and re-run the reindex with a dry run first. Because the continuity HTTP endpoint is not yet implemented (SP-002), verify continuity in-process via the IEvidenceReindexService.VerifyContinuityAsync library API rather than the CLI verb.

  2. Anchoring warnings: check.evidencelocker.merkle inspects an anchors/ directory under the locker path when anchoring is enabled.

FLAG — Draft/roadmap: The Doctor merkle check’s remediation text suggests stella evidence anchor create, but no stella evidence anchor command exists in the CLI (the only anchor verb is stella evidence proof anchor, which anchors a proof to a Rekor transparency log). A standalone “Merkle anchor chain” with create/verify/export/recover/new-chain operations is NOT IMPLEMENTED. Bundle Merkle roots are computed at build time (MerkleTreeCalculator) and time anchoring is via RFC3161 timestamps on signatures; periodic anchoring of the whole locker into a transparency log is forward-looking. Do not script against stella evidence anchor *.

INC-004: Storage Full

Symptoms:

Immediate Actions:

# Inspect host/object-store usage with platform tooling (df, S3 console/CLI).
# There is no `stella evidence storage stats|analyze` command.

# Confirm whether the retention sweep is enabled and what it would remove (dry run via config):
#   EvidenceLocker:RetentionSweep:Enabled = true
#   EvidenceLocker:RetentionSweep:DryRun  = true   # logs candidates without deleting
stella config show EvidenceLocker

Resolution:

  1. Enable / tune the retention sweep: set EvidenceLocker:RetentionSweep:Enabled=true, adjust DefaultRetentionDays / Interval, restart the service, and review which capsules are swept (run with DryRun=true first). Capsules under a legal hold or carrying a regulatory retention record are not swept.

  2. Expand storage: grow the FileSystem:RootPath volume, or expand the S3 bucket/quota.

FLAG — NOT IMPLEMENTED: stella evidence cleanup, stella evidence archive, and stella evidence storage * were in the earlier draft but do not exist. Storage reclamation is driven by the configurable retention sweep, not by an operator cleanup verb.

Disaster Recovery

Recovery is performed at the infrastructure layer (PostgreSQL snapshot/restore + object-store restore), not via stella evidence restore / stella backup (those commands do not exist). See the migration-recovery runbook for the forward-only DB recovery model (restore from a PostgreSQL snapshot rather than downgrading).

DR-001: Full Evidence Locker Recovery

Prerequisites:

Procedure:

  1. Restore the database from the most recent good snapshot (PostgreSQL-level restore; the service auto-migrates forward on startup).

  2. Restore the object store:

    • FileSystem: restore the RootPath volume from its snapshot/backup.
    • AmazonS3: restore from bucket versioning/replication.

  3. Point the service at the restored stores via configuration (EvidenceLocker:Database:ConnectionString, EvidenceLocker:ObjectStore:*) and restart.

  4. Verify:

    # Run the per-module Doctor checks by id (--category "Evidence Locker" does NOT work; the
# category is `Security` — see Pre-flight CORRECTION):
stella doctor --check check.evidencelocker.retrieval
stella doctor --check check.evidencelocker.index
stella doctor --check check.evidencelocker.provenance
# Spot-check a known bundle end-to-end:
stella evidence export <known-bundle-id> --output recovered.tar.gz
stella evidence verify recovered.tar.gz

DR-002: Post-Reindex / Post-Upgrade Continuity

If a recovery follows a reindex or schema migration, confirm chain-of-custody continuity between the pre- and post-change Merkle roots:

# NOTE: the CLI verb below posts to the unmapped /api/v1/evidence/continuity/verify endpoint
# (SP-002 NOT IMPLEMENTED). Until that route ships, verify continuity in-process via
# IEvidenceReindexService.VerifyContinuityAsync rather than this CLI command.
stella evidence verify-continuity --old-root sha256:<old> --new-root sha256:<new> --output table

A failing continuity result means the recovered state is not provably continuous with the prior state — restore the earlier snapshot instead.

Offline Mode Operations

Preparing an Offline Evidence Pack

# Export a bundle as a portable tarball for offline audit
stella evidence export <bundle-id> --output evidence-pack.tar.gz [--include-layers] [--include-rekor-proofs]

# Or export a single-file, signed evidence card
stella evidence card export <pack-id> --output pack.evidence-card.json

Verifying Evidence Offline

# Verify a bundle without contacting the Rekor transparency log
stella evidence verify evidence-pack.tar.gz --offline

# Verify an evidence card offline, optionally against a local trust root
stella evidence card verify pack.evidence-card.json --offline --trust-root /path/to/trust-root

# Replay a deterministic verdict from a stored capsule/pack
stella evidence replay run --artifact <artifact-digest>
stella evidence replay score --pack <pack-id>

FLAG — Draft/roadmap: stella evidence replay, proof, provenance show, and seal are registered subcommands but currently emit illustrative/sample output rather than calling a backend. Server-side capsule replay is exposed at POST /api/v1/evidence/capsules/{id}/replay; prefer the API for authoritative replay until the CLI verbs are wired to it.

Monitoring

FLAG — NOT IMPLEMENTED: There is no committed “Evidence Locker” Grafana dashboard and no stella_evidence_* meter (see “Metrics to Watch”). Monitor via the Doctor Evidence Locker checks (run on a schedule through the Doctor scheduler) plus standard /readyz and host disk/storage signals. Add this section’s dashboard reference once a meter and dashboard exist.

Scheduled health coverage (Doctor):

Evidence Capture (for incidents)

NOTE: There is no stella evidence diagnostics command. Collect incident evidence manually. Also note (CORRECTION 2026-05-31): there is no stella logs <service> command and no stella doctor --category "Evidence Locker" — the commands below use the real surfaces (stella obs logs ... --service evidence-locker and per-check Doctor ids).

# Service logs (platform log reader lives under `obs`; requires --from/--to, supports --service):
stella obs logs --service evidence-locker \
  --from "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --to   "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --output ndjson > /tmp/evidence-locker.log

# Health snapshot (per-check ids; --category "Evidence Locker" runs zero checks — use Security or ids):
stella doctor --check check.evidencelocker.retrieval  >  /tmp/evidence-doctor.txt
stella doctor --check check.evidencelocker.index      >> /tmp/evidence-doctor.txt
stella doctor --check check.evidencelocker.provenance >> /tmp/evidence-doctor.txt
stella doctor --check check.evidencelocker.merkle     >> /tmp/evidence-doctor.txt

# Readiness probe output
curl -fsS http://<evidence-locker-host>/readyz > /tmp/evidence-readyz.txt

Attach: the relevant stella evidence verify --output json result, the failing bundle/pack id, the Doctor output above, and the service logs.

API & Scope Reference (verified against source)

Authentication uses the canonical Authority scopes (src/Authority/.../StellaOpsScopes.cs); WebService policies are registered in StellaOpsResourceServerPolicies.cs via AddStellaOpsScopePolicy(EvidenceRead, …) etc. The policy name and the scope claim value are identical here (the EvidenceRead/EvidenceCreate/ EvidenceHold constants equal evidence:read/evidence:create/evidence:hold):

Scope (claim value)Policy constantEndpoints that require it (verified)
evidence:readEvidenceReadRead: GET /evidence/score, /evidence/{id} + /chain//download//portable, POST /evidence/verify, packs/proofs/audit/receipts/thread, verdict reads + /verify + /envelope, capsule reads + /verify, regulatory artifact-ledger
evidence:createEvidenceCreatePOST /evidence (gate artifact), POST /evidence/hold/{caseId} (legal hold creation), POST /evidence/qa/.../readback-import, POST /api/v1/verdicts, capsule POST /, /{id}/seal, /{id}/export, /{id}/replay, POST /tlpt-packs
evidence:holdEvidenceHoldPOST /evidence/snapshot (only this route uses the hold policy)

CORRECTION (2026-05-31): The earlier table mapped evidence:create to “snapshots” and evidence:hold to “place/release legal holds (/evidence/snapshot)”. That is inverted in the code: POST /evidence/snapshot is the only route gated by EvidenceHold, whereas the actual legal-hold creation route POST /evidence/hold/{caseId} is gated by EvidenceCreate. The table above reflects the verified .RequireAuthorization(...) calls in Program.cs and the Api/ endpoint files.

Representative WebService routes (not exhaustive — verify against src/EvidenceLocker/StellaOps.EvidenceLocker/Api and the WebService Program.cs):

FLAG — NOT IMPLEMENTED (server side): /api/v1/evidence/{reindex,continuity,migrate}/... are referenced by the CLI but have no server-side route registration anywhere in src/. They are listed here only to document the gap; do not treat them as live endpoints (see SP-002/SP-003).

Escalation Path

  1. L1 (On-call): bundle verify, holds list/show, Doctor checks, log capture
  2. L2 (Platform team): re-index, retention/storage tuning, object-store issues
  3. L3 (Architecture): schema migration/rollback, DR (DB + object-store restore), continuity

_Last reconciled: 2026-05-31 (UTC). Source of truth: src/EvidenceLocker (WebService Program.cs