Runbook: Replay Operations

This runbook governs day-to-day deterministic replay operations — ingestion, verification, retention, and incident handling — across online and air-gapped Stella Ops environments. Deterministic replay lets an operator reconstruct any past scan decision from its signed inputs and outputs, which is the backbone of the platform’s verifiable-evidence guarantee.

Audience: Ops Guild, Evidence Locker Guild, Scanner Guild, Authority/Signer, Attestor.

Read first:

1. Terminology

2. Normal operations

  1. Ingestion
    • Scanner WebService writes manifest metadata to replay_runs.
    • Bundles are uploaded to CAS (cas://replay/...) and mirrored into Evidence Locker (evidence.replay_bundles).
    • Authority triggers DSSE signing; Attestor optionally anchors to Rekor.
  2. Verification
    • A nightly job runs stella verify on the latest N replay manifests per tenant.
    • Metrics replay_verify_total{result} and replay_bundle_size_bytes are recorded in the Telemetry Stack (see docs/modules/telemetry/architecture.md).
    • Failures alert #ops-replay via PagerDuty with a runbook link.
  3. Retention
    • Hot CAS retention: 180 days (configurable per tenant). The replay-retention cron job prunes expired digests and writes audit entries.
    • Cold storage (Evidence Locker): 2 years; legal holds extend via /evidence/holds. Record holds in timeline.events with type replay.hold.created.
    • Retention declaration: validate against the frozen retention schema (see docs/modules/replay/guides/retention-schema-freeze-2025-12-10.md, frozen 2025-12-10). Include retention_policy_id, tenant_id, bundle_type, retention_days, legal_hold, purge_after, checksum, and created_at. Audit the checksum via DSSE envelope when persisting.
  4. Access control
    • Only service identities with the replay:read scope may fetch bundles. The CLI requires a device or client-credential flow with DPoP.

3. Incident response (replay integrity)

StepActionOwnerNotes
1Page Ops via the replay_verify_total{result="failed"} alertObservabilityInclude scan ID, tenant, and failure codes
2Lock affected bundles (POST /evidence/holds)Evidence LockerReference the incident ticket
3Re-run stella verify with --explain to gather diffsScanner GuildAttach the diff JSON to the incident
4Check Rekor inclusion proofs (stella verify --ledger)AttestorFlag if the ledger is mismatched or stale
5On tool-hash drift, coordinate Signer for rotationAuthority/SignerRotate the DSSE profile, update RootPack
6Update the incident timeline (this runbook → Incident log)Ops GuildRecord timestamps and decisions
7Close the hold once resolved, publish the postmortemOps + DocsPostmortem must reference the replay spec sections

4. Air-gapped workflow

  1. Receive the Offline Kit bundle, containing:
    • offline/replay/<scan-id>/manifest.json
    • Bundles + DSSE signatures
    • A RootPack snapshot
  2. Run stella replay manifest.json --strict --offline using the local CLI.
  3. Load feed/policy snapshots from the kit; never reach external networks.
  4. Store verification logs under ops/offline/replay/<scan-id>/.
  5. Sync results back to Evidence Locker once connectivity is restored.

5. Maintenance checklist

6. Incident log

Record each replay-integrity incident and drill here.

Date (UTC)Incident IDTenantSummaryFollow-up
No incidents recorded yet.

7. References

Created 2025-11-03. Update alongside replay operational changes and after each integrity drill.