Evidence Locker Operations Runbook

Audience: Platform and on-call engineers operating the Stella Ops Evidence Locker — the tamper-evident store for release evidence bundles, decision capsules, verdict attestations, and the regulatory artifact ledger. Use this runbook to verify bundles, manage retention and legal holds, and recover the locker after a failure.

Status: RECONCILED AGAINST SOURCE (2026-08-22 UTC)

Source-composition note (2026-08-22): the six score/snapshot/UUID/QA/verify/hold handlers now come from StellaOps.EvidenceLocker.Application/Api/EvidenceCoreEndpoints.cs. The live predecessor still serves the unchanged /evidence/** paths; the equivalent /api/evidence/v1/evidence/** routes exist only in the source-only evidence-web target and do not receive traffic. Do not use those target paths until the owner-approved EVD-9 window applies the route swap. This day-to-day runbook is not the complete EVD-9 cutover procedure; the ordered backup/copy/parity/hash-chain/repoint/revoke/rollback/soak/forcing runbook and its approvals remain a NO-GO gate for that window.

Reconciliation note (2026-05-31): This runbook was reconciled against src/EvidenceLocker, the CLI evidence command groups (src/Cli/StellaOps.Cli/Commands/EvidenceCommandGroup.cs, EvidenceHoldsCommandGroup.cs), the Doctor Evidence Locker plugin (src/Doctor/__Plugins/StellaOps.Doctor.Plugin.EvidenceLocker), the Doctor engine category filter (src/__Libraries/StellaOps.Doctor/Engine/CheckRegistry.cs), the WebService route map (src/EvidenceLocker/.../StellaOps.EvidenceLocker.WebService/Program.cs + the Api/ endpoint files), and the canonical scope catalog (src/Authority/.../StellaOpsScopes.cs). Many commands in the earlier draft (stella evidence index|anchor|storage|cleanup|archive|verify-report| inspect|recover|mark-corrupted|exists|restore|diagnostics, stella backup, stella config set, stella service restart) and the four stella_evidence_* / stella_merkle_anchor_age_seconds metrics do not exist in the codebase and have been corrected, removed, or flagged NOT IMPLEMENTED below.

A second reconciliation pass (2026-05-31) found additional drift the first pass missed and corrected it inline: (a) stella doctor --category "Evidence Locker" does not work — the category filter only matches the DoctorCategory enum (Enum.TryParse), and the Evidence Locker plugin’s category is Security, while “Evidence Locker” is only its display name; use --check <id> or --category Security; (b) there is no stella logs <service> command — the platform log reader is stella obs logs --from <iso> --to <iso> --service evidence-locker (no positional service, no --since); © the reindex / verify-continuity / migrate CLI verbs targeted /api/v1/evidence/{reindex,continuity,migrate}/... endpoints that are not implemented server-side (the reindex service exists as a library only, IEvidenceReindexService, never HTTP-mapped), and those three verbs were deleted on 2026-09-03 for that reason (SPRINT_20260903_005, EVD-DEL-2); (d) the live evidence holds CLI is the simpler one wired in EvidenceCommandGroup (list, create --reason, release <hold-id>) — the richer EvidenceHoldsCommandGroup (--name/--scope/--digest, show, --confirm) is present in the tree but not wired into the command factory. The code is the source of truth; this doc has been brought in line with it.

Scope

Evidence locker operations: evidence bundle export/verification, decision-capsule lifecycle, verdict attestation storage, regulatory artifact ledger, retention sweeping, evidence re-indexing, and disaster recovery. The service is HTTP/CLI-driven; there is no bespoke “anchor chain” or “index maintenance” subsystem (see flagged items).

Pre-flight Checklist

Environment Verification

# Run the Evidence Locker health checks (Doctor plugin). Prefer running the
# individual checks by id — the plugin's Doctor *category* is `Security`
# (its display name is "Evidence Locker"), so a category filter must use the
# enum value, not the display name (see CORRECTION below).
stella doctor --check check.evidencelocker.retrieval    # Attestation Retrieval
stella doctor --check check.evidencelocker.index        # Evidence Index Consistency
stella doctor --check check.evidencelocker.provenance   # Provenance Chain Integrity
stella doctor --check check.evidencelocker.merkle       # Merkle Anchor Verification (only runs if EvidenceLocker:Anchoring:Enabled=true)

# Category filter (runs ALL Security-category checks, not just Evidence Locker):
stella doctor --category Security

# Liveness / readiness probes exposed by the WebService (no auth required):
curl -fsS http://<evidence-locker-host>/healthz       # liveness
curl -fsS http://<evidence-locker-host>/readyz        # readiness
curl -fsS http://<evidence-locker-host>/health/ready  # legacy readiness alias

CORRECTION (2026-05-31): stella doctor --category "Evidence Locker" does not work. The CLI passes --category to DoctorRunOptions.Categories, which is matched with Enum.TryParse<DoctorCategory>(...) in CheckRegistry.GetFilteredPlugins — only enum names (Core, Database, Security, Cryptography, Attestation, Docker, …) match. The Evidence Locker plugin declares Category => DoctorCategory.Security and DisplayName => "Evidence Locker"; passing the display name parses to no category, so zero checks run. Use the per-check --check check.evidencelocker.* ids (recommended — they target exactly this module) or --category Security (broader; runs every Security-category plugin).

NOTE: There is no stella evidence status, stella evidence index status, or stella evidence anchor verify command. Health and index/anchor consistency are surfaced through the Doctor checks above, not through dedicated evidence subcommands.

Metrics to Watch

FLAG — NOT IMPLEMENTED: The earlier draft listed stella_evidence_artifacts_total, stella_evidence_retrieval_latency_seconds, stella_evidence_storage_bytes, and stella_merkle_anchor_age_seconds. None of these metric names exist anywhere in src/ or devops/, and there is no “Evidence Locker” Grafana dashboard committed in the repo. Until the service emits a documented meter, monitor Evidence Locker health via:

  • the Doctor checks listed above (run on a schedule via the Doctor scheduler), and
  • the standard ASP.NET / /readyz health probes and host-level disk/storage metrics.

When dedicated metrics land, replace this block with the real meter and instrument names (and link the dashboard JSON under devops/).

Standard Procedures

SP-001: Evidence Bundle Verification

Frequency: On-demand (e.g. before an audit handoff), or after import on a downstream host Duration: Seconds to minutes depending on bundle size

The CLI verifies an exported bundle archive (it does not run a server-side “integrity sweep”).

  1. Export the bundle you want to check (or use one already exported):

    stella evidence export <bundle-id> --output bundle.tar.gz
# e.g. bundle-id = eb-2026-01-06-abc123

  2. Verify the exported archive (checksums, manifest, DSSE envelope structure, Rekor proofs):

    # Full verification
stella evidence verify bundle.tar.gz

# Air-gapped: skip Rekor transparency-log lookups
stella evidence verify bundle.tar.gz --offline

# Checksums + manifest only (skip DSSE signature structure check)
stella evidence verify bundle.tar.gz --skip-signatures

# Machine-readable output
stella evidence verify bundle.tar.gz --output json

    This legacy reader reports cryptographic authenticity as unverified and returns nonzero; its structural checks cannot authenticate a signature or transparency proof.

  3. Single-file card export is unavailable: its former CLI endpoint had no owner implementation. Existing cards can be inspected with stella evidence card verify <file>, but authenticity remains unverified and the command returns nonzero. The unused --trust-root option is removed.

NOTE: stella evidence verify --mode quick|full and stella evidence verify-report do not exist. verify takes a bundle path and emits a pass/fail table (or --output json).

SP-002: Evidence Re-Index

Frequency: After a schema or hashing-algorithm change to evidence bundles Duration: Varies with record count

Re-indexing recomputes Merkle roots for stored bundles.

There is no operator command for this procedure. The reindex engine is a library — IEvidenceReindexService / EvidenceReindexService, with ReindexAsync / VerifyContinuityAsync / GenerateCrossReferenceAsync / CreateCheckpointAsync / RollbackToCheckpointAsync — registered in the Evidence Locker’s DI container and covered by EvidenceReindexServiceTests / EvidenceReindexIntegrationTests. It has never been wired to a route: no Map* registration for /api/v1/evidence/reindex/* or /api/v1/evidence/continuity/verify exists anywhere in src/, on the Evidence Locker WebService or on the consolidated Attestor host.

The evidence reindex and evidence verify-continuity CLI verbs were deleted on 2026-09-03 (SPRINT_20260903_005, EVD-DEL-2) precisely because they could only ever fail against those unmapped routes. Drive the engine in-process or from a maintenance task; a CLI surface returns only when an owner route exists. Re-index is rarely needed in steady state — the DB auto-migrates forward, see SP-003.

SP-003: Schema Migration

Frequency: During upgrades that change the evidence schema version Duration: Startup convergence; varies with the migration set

There is no operator command for this procedure either. As with SP-002, the migrate CLI verb posted to /api/v1/evidence/migrate/{plan,execute,rollback}, which no host maps, so it was deleted on 2026-09-03 (SPRINT_20260903_005, EVD-DEL-2) together with its --dry-run, --from-version, --to-version and --rollback flags.

The authoritative migration path is automatic: the service-owned PostgreSQL schema auto-migrates on startup (EvidenceLockerMigrationHostedService, migrations 001–012 under StellaOps.EvidenceLocker.Infrastructure/Db/Migrations — verified: 001_initial_schema … 012_capsule_retention_sweep) when EvidenceLocker:Database:ApplyMigrationsAtStartup is true (the default). Platform migration policy is forward-only (ADR-004); recover DDL state via PostgreSQL snapshot restore — see migration-recovery runbook. There is no evidence-schema rollback verb at all — the deleted --rollback flag posted to /api/v1/evidence/migrate/rollback, which no host maps.

Frequency: Background (configurable); holds managed on-demand Duration: Varies

Retention is enforced by a background sweep, not a stella evidence cleanup command.

CORRECTION (2026-05-31): The earlier draft documented holds list [--status], holds create --name --scope --digest --component --from --to, holds show [--artifacts], and holds release --confirm --reason. Those flags belong to a SECOND, richer implementation (EvidenceHoldsCommandGroup) that exists in the source tree but is not wired into the command factory — CommandFactory.cs builds EvidenceCommandGroup.BuildEvidenceCommand, which adds the local simple BuildHoldsCommand (only list, create --reason, release <hold-id>; no show subcommand). Script against the minimal surface above until the richer group is wired in.

FLAG — Draft/roadmap: even the wired holds CLI currently renders sample/illustrative output and is not connected to a live holds API in the WebService. Treat hold IDs and counts in its output as placeholders. The real server-side legal-hold creation endpoint is POST /evidence/hold/{caseId} (requires evidence:create; see API reference) and storage-layer hold enforcement is real for S3 backends via Object Lock legal hold (see SP-005).

SP-005: Object Store & WORM / Object Lock

Frequency: Configuration-time; review during audits Duration: N/A

The evidence object store backend is selected by EvidenceLocker:ObjectStore:Kind. Kind is [Required] (enum FileSystem | AmazonS3) — there is no code-level default; the on-prem deployment config sets FileSystem:

Per repo policy, on-prem FileSystem is the default; do not switch to a cloud-managed backend as a default. To inspect effective configuration:

stella config list
stella config show EvidenceLocker

NOTE: stella config set does not exist — the CLI config group is read-only (list, show). Change configuration through the service’s config sources (env / files / DB overrides) and restart the service.

Incident Procedures

INC-001: Bundle Verification Failure

Symptoms:

Investigation:

# Re-run with JSON to capture which check failed
stella evidence verify bundle.tar.gz --output json > /tmp/verify.json

# Inspect the bundle / proof chain via the API (requires evidence:read)
curl -fsS -H "Authorization: Bearer $TOKEN" \
  http://<evidence-host>/api/evidence/v1/evidence/<bundleId>            # bundle metadata
curl -fsS -H "Authorization: Bearer $TOKEN" \
  http://<evidence-host>/api/evidence/v1/evidence/<bundleId>/chain      # custody / proof chain

Resolution:

  1. Transport corruption (re-export fixes it): re-run stella evidence export <bundle-id> and re-verify. The on-disk bundle in the locker is the source of truth.

  2. Checksum/manifest mismatch in the stored bundle: run the Doctor checks (check.evidencelocker.index, check.evidencelocker.provenance) to confirm whether the stored artifact or only the export is affected, then restore from backup (DR-001).

  3. Suspected false positive (tooling bug): verify with --skip-signatures to isolate the failing layer, capture the JSON output, and escalate to L2/L3.

INC-002: Evidence Retrieval Failure

Symptoms:

Investigation:

# Retrieval health check (probes the locker's ability to read attestation artifacts)
stella doctor --check check.evidencelocker.retrieval

# Index consistency between the index and stored artifacts
stella doctor --check check.evidencelocker.index

# Direct API probes (evidence:read)
curl -fsS -H "Authorization: Bearer $TOKEN" \
  http://<evidence-locker-host>/api/v1/evidence/packs
curl -fsS -H "Authorization: Bearer $TOKEN" \
  http://<evidence-locker-host>/api/v1/evidence/packs/<id>

Resolution:

  1. Object store unreachable: verify the configured EvidenceLocker:ObjectStore backend (filesystem mount present and writable, or S3 endpoint/credentials reachable). Check the service logs (stella obs logs --service evidence-locker --from <iso8601> --to <iso8601>) and /readyz. (CORRECTION: there is no stella logs <service> command; the platform log reader lives under obs and requires --from/--to rather than --since — see the note in the Evidence Capture section.)

  2. Index inconsistency: if check.evidencelocker.index flags drift, run a re-index (SP-002) after taking a backup. There is no stella evidence index rebuild command.

  3. Database / migration issue: confirm the schema converged on startup (migrations 001–012); a missing table is the most common cause of 5xx (see AGENTS.md §2.7).

INC-003: Merkle Anchor / Continuity Concern

Symptoms:

Investigation:

# Merkle/anchor health (no-op/skip unless anchoring is enabled)
stella doctor --check check.evidencelocker.merkle

Continuity across roots has no CLI verb (SP-002): call IEvidenceReindexService.VerifyContinuityAsync in-process against the old and new roots.

Resolution:

  1. Continuity failure after reindex: treat the new root as unverified. Restore the pre-reindex state from backup (DR-001) before recomputing. Verify continuity in-process via the IEvidenceReindexService.VerifyContinuityAsync library API — there is no CLI verb for it.

  2. Anchoring warnings: check.evidencelocker.merkle inspects an anchors/ directory under the locker path when anchoring is enabled.

FLAG — Draft/roadmap: The Doctor merkle check’s remediation text suggests stella evidence anchor create, but no stella evidence anchor command exists in the CLI (the only anchor verb is stella evidence proof anchor, which anchors a proof to a Rekor transparency log). A standalone “Merkle anchor chain” with create/verify/export/recover/new-chain operations is NOT IMPLEMENTED. Bundle Merkle roots are computed at build time (MerkleTreeCalculator) and time anchoring is via RFC3161 timestamps on signatures; periodic anchoring of the whole locker into a transparency log is forward-looking. Do not script against stella evidence anchor *.

INC-004: Storage Full

Symptoms:

Immediate Actions:

# Inspect host/object-store usage with platform tooling (df, S3 console/CLI).
# There is no `stella evidence storage stats|analyze` command.

# Confirm whether the retention sweep is enabled and what it would remove (dry run via config):
#   EvidenceLocker:RetentionSweep:Enabled = true
#   EvidenceLocker:RetentionSweep:DryRun  = true   # logs candidates without deleting
stella config show EvidenceLocker

Resolution:

  1. Enable / tune the retention sweep: set EvidenceLocker:RetentionSweep:Enabled=true, adjust DefaultRetentionDays / Interval, restart the service, and review which capsules are swept (run with DryRun=true first). Capsules under a legal hold or carrying a regulatory retention record are not swept.

  2. Expand storage: grow the FileSystem:RootPath volume, or expand the S3 bucket/quota.

FLAG — NOT IMPLEMENTED: stella evidence cleanup, stella evidence archive, and stella evidence storage * were in the earlier draft but do not exist. Storage reclamation is driven by the configurable retention sweep, not by an operator cleanup verb.

Disaster Recovery

Recovery is performed at the infrastructure layer (PostgreSQL snapshot/restore + object-store restore), not via stella evidence restore / stella backup (those commands do not exist). See the migration-recovery runbook for the forward-only DB recovery model (restore from a PostgreSQL snapshot rather than downgrading).

DR-001: Full Evidence Locker Recovery

Prerequisites:

Procedure:

  1. Restore the database from the most recent good snapshot (PostgreSQL-level restore; the service auto-migrates forward on startup).

  2. Restore the object store:

    • FileSystem: restore the RootPath volume from its snapshot/backup.
    • AmazonS3: restore from bucket versioning/replication.

  3. Point the service at the restored stores via configuration (EvidenceLocker:Database:ConnectionString, EvidenceLocker:ObjectStore:*) and restart.

  4. Verify:

    # Run the per-module Doctor checks by id (--category "Evidence Locker" does NOT work; the
# category is `Security` — see Pre-flight CORRECTION):
stella doctor --check check.evidencelocker.retrieval
stella doctor --check check.evidencelocker.index
stella doctor --check check.evidencelocker.provenance
# Spot-check a known bundle end-to-end:
stella evidence export <known-bundle-id> --output recovered.tar.gz
stella evidence verify recovered.tar.gz

DR-002: Post-Reindex / Post-Upgrade Continuity

If a recovery follows a reindex or schema migration, confirm chain-of-custody continuity between the pre- and post-change Merkle roots. There is no CLI verb for this (SP-002): call IEvidenceReindexService.VerifyContinuityAsync in-process with the old and new roots.

A failing continuity result means the recovered state is not provably continuous with the prior state — restore the earlier snapshot instead.

Offline Mode Operations

Preparing an Offline Evidence Pack

# Export a bundle as a portable tarball for offline audit
stella evidence export <bundle-id> --output evidence-pack.tar.gz [--include-layers] [--include-rekor-proofs]

Verifying Evidence Offline

# Verify a bundle without contacting the Rekor transparency log
stella evidence verify evidence-pack.tar.gz --offline

# Inspect a card offline; this reports unverified authenticity and returns nonzero
stella evidence card verify pack.evidence-card.json --offline

# Replay a deterministic verdict from a stored capsule/pack
stella evidence replay run --artifact <artifact-digest>
stella evidence replay score --pack <pack-id>

FLAG — Draft/roadmap: stella evidence replay, proof, provenance show, and seal are registered subcommands but currently emit illustrative/sample output rather than calling a backend. Server-side capsule replay is exposed at POST /api/v1/evidence/capsules/{id}/replay; prefer the API for authoritative replay until the CLI verbs are wired to it.

Monitoring

FLAG — NOT IMPLEMENTED: There is no committed “Evidence Locker” Grafana dashboard and no stella_evidence_* meter (see “Metrics to Watch”). Monitor via the Doctor Evidence Locker checks (run on a schedule through the Doctor scheduler) plus standard /readyz and host disk/storage signals. Add this section’s dashboard reference once a meter and dashboard exist.

Scheduled health coverage (Doctor):

Evidence Capture (for incidents)

NOTE: There is no stella evidence diagnostics command. Collect incident evidence manually. Also note (CORRECTION 2026-05-31): there is no stella logs <service> command and no stella doctor --category "Evidence Locker" — the commands below use the real surfaces (stella obs logs ... --service evidence-locker and per-check Doctor ids).

# Service logs (platform log reader lives under `obs`; requires --from/--to, supports --service):
stella obs logs --service evidence-locker \
  --from "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --to   "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --output ndjson > /tmp/evidence-locker.log

# Health snapshot (per-check ids; --category "Evidence Locker" runs zero checks — use Security or ids):
stella doctor --check check.evidencelocker.retrieval  >  /tmp/evidence-doctor.txt
stella doctor --check check.evidencelocker.index      >> /tmp/evidence-doctor.txt
stella doctor --check check.evidencelocker.provenance >> /tmp/evidence-doctor.txt
stella doctor --check check.evidencelocker.merkle     >> /tmp/evidence-doctor.txt

# Readiness probe output
curl -fsS http://<evidence-locker-host>/readyz > /tmp/evidence-readyz.txt

Attach: the relevant stella evidence verify --output json result, the failing bundle/pack id, the Doctor output above, and the service logs.

API & Scope Reference (verified against source)

Authentication uses the canonical Authority scopes (src/Authority/.../StellaOpsScopes.cs); WebService policies are registered in StellaOpsResourceServerPolicies.cs via AddStellaOpsScopePolicy(EvidenceRead, …) etc. The policy name and the scope claim value are identical here (the EvidenceRead/EvidenceCreate/ EvidenceHold constants equal evidence:read/evidence:create/evidence:hold):

Scope (claim value)Policy constantEndpoints that require it (verified)
evidence:readEvidenceReadRead: GET /api/evidence/v1/evidence/score, /api/evidence/v1/evidence/{id} + /chain//download//portable, /api/evidence/v1/evidence/by-release/{releaseId}/portable, POST /api/evidence/v1/evidence/verify, packs/proofs/audit/receipts/thread, verdict reads + /verify + /envelope, capsule reads + /verify, regulatory artifact-ledger
evidence:createEvidenceCreatePOST /api/evidence/v1/evidence/hold/{caseId} (legal hold creation), POST /api/evidence/v1/evidence/qa/.../readback-import, POST /api/v1/verdicts, capsule POST /, /{id}/seal, /{id}/export, /{id}/replay, POST /tlpt-packs
evidence:holdEvidenceHoldPOST /api/evidence/v1/evidence/snapshot (only this route uses the hold policy)

EVD-10a retired evidence-locker-web, so /api/evidence/v1/evidence/** is the only served form: the policies, methods and endpoint names above are the consolidated host’s, and generated bundle/hold links carry that prefix. POST /evidence (gate artifact) was withdrawn with the predecessor host rather than carried — it had no consumers; its read side GET /evidence/score survives at the consolidated prefix.

CORRECTION (2026-05-31): The earlier table mapped evidence:create to “snapshots” and evidence:hold to “place/release legal holds (/evidence/snapshot)”. That is inverted in the code: POST /api/evidence/v1/evidence/snapshot is the only route gated by EvidenceHold, whereas the actual legal-hold creation route POST /api/evidence/v1/evidence/hold/{caseId} is gated by EvidenceCreate. The table above reflects the verified .RequireAuthorization(...) calls in the consolidated host’s Program.cs and the shared Api/ endpoint files.

Representative WebService routes (not exhaustive — verify against src/Evidence/__Libraries/StellaOps.EvidenceLocker.Application/Api and the consolidated host’s src/Evidence/StellaOps.Evidence.WebService/Program.cs):

EVD-10a retired evidence-locker-web and its inline routes. These are now operator targets on the consolidated Evidence host under /api/evidence/v1. POST /evidence (gate-artifact store) was withdrawn rather than carried.

NOT SERVED: /api/v1/evidence/{reindex,continuity,migrate}/... have no route registration anywhere in src/. The CLI verbs that referenced them were deleted on 2026-09-03, so nothing calls them any more; they are named here only so the gap stays visible (see SP-002/SP-003).

Escalation Path

  1. L1 (On-call): bundle verify, holds list/show, Doctor checks, log capture
  2. L2 (Platform team): re-index, retention/storage tuning, object-store issues
  3. L3 (Architecture): schema migration/rollback, DR (DB + object-store restore), continuity

_Last reconciled: 2026-05-31 (UTC). Source of truth: src/EvidenceLocker (WebService Program.cs