Replay Determinism Test

Audience: QA engineers and operators verifying deterministic replay.

This runbook verifies that two fixture or replay runs produce identical canonical server-side artefact hashes — the core determinism guarantee behind Stella Ops’s verifiable evidence. It drives the Tester service compare endpoint:

POST /api/v1/tester/replay-and-compare

The endpoint is bearer-gated and forwards the caller’s Authorization and X-StellaOps-TenantId headers to every downstream service. It calls only in-stack public REST APIs; it does not read module databases or persistence libraries.

Where authorization is actually enforced. The Tester mints no token of its own and enforces no local scope — it relays your bearer, and each upstream applies its own canonical-catalog scope. So the token you present is the only authority in play: the oracle can never read anything you could not read yourself. A credential-less request is rejected at the Tester (401 authorization_required) before any downstream call, which is what stops the compose-network bypass being laundered into an anonymous read.

Prerequisites

• Tester service reachable through the gateway at https://stella-ops.local.
• Bearer token carrying findings:read— the scope the Findings ledger list requires (StellaOpsScopes.FindingsRead; compose binds it via findings__ledger__Authority__RequiredScopes__0). That is the only scope needed today, because findings is the only measurable scope. evidence:read / timeline:read become relevant only if those upstreams gain a runId filter.
• Tenant header value, normally default for local compose.
• Two run identifiers produced by the same deterministic fixture import or replay process.

Back-to-Back Compare

1. Seed two runs with the same fixture input. Use the existing operator harness or Tester fixture endpoints, for example:

   & tools/scripts/qa/Invoke-OperatorScenarios.ps1 -Scenario topo-103-replay-run-1
   & tools/scripts/qa/Invoke-OperatorScenarios.ps1 -Scenario topo-103-replay-run-2
   

2. Capture the run identifiers from the harness output. The Tester service treats them as opaque strings and passes them downstream as runId.

3. Run the compare. findings is currently the only scope that yields a determinism proof — it is the only one whose upstream actually filters by runId (see the scope table below). Ask for it alone:

   curl -k -X POST "https://stella-ops.local/api/v1/tester/replay-and-compare" \
     -H "Authorization: Bearer $TOKEN" \
     -H "X-StellaOps-TenantId: default" \
     -H "Content-Type: application/json" \
     -d '{
       "runIdA": "advanced-assurance-golden-2026-05-18T13:00Z",
       "runIdB": "advanced-assurance-golden-2026-05-18T13:30Z",
       "scope": ["findings"]
     }'
   

4. Expected result for identical input:

   {
     "verdict": "identical",
     "runIdA": "advanced-assurance-golden-2026-05-18T13:00Z",
     "runIdB": "advanced-assurance-golden-2026-05-18T13:30Z",
     "perScopeDiffs": [
       {
         "scope": "findings",
         "verdict": "identical",
         "runIdFilteringSupported": true,
         "sha256A": "sha256:...",
         "sha256B": "sha256:...",
         "rowCountA": 7,
         "rowCountB": 7
       }
     ]
   }
   

If any scope drifts, the top-level verdict becomes drift, and the drifting scope includes diff.onlyInA, diff.onlyInB, and diff.changed.

Asking for a scope that cannot be measured

You may still request the other three scopes, but the oracle will not pretend to measure them. It reports indeterminate — no digest, no row count, and the overall verdict is dragged off identical so a green can never be read from an unmeasured run:

{
  "verdict": "indeterminate",
  "perScopeDiffs": [
    { "scope": "findings", "verdict": "identical", "runIdFilteringSupported": true, "sha256A": "sha256:…", "sha256B": "sha256:…", "rowCountA": 7, "rowCountB": 7 },
    { "scope": "evidence-packs", "verdict": "indeterminate", "runIdFilteringSupported": false,
      "service": "EvidenceLocker",
      "reason": "EvidenceLocker GET /api/v1/evidence/packs accepts no runId parameter; it lists every pack for the tenant…" }
  ]
}

Why this changed (SPRINT_20260712_008 / W3-E). Before this fix the oracle fetched every scope regardless. evidence-packs therefore compared the same unfiltered tenant-wide pack list against itself and reported identical for any two run ids — a determinism proof it never performed. timeline-events and linksets hit routes that do not exist, returned 404, and sank the entire call to 404 run_not_found — so the four-scope invocation this runbook used to recommend failed for every operator who ran it, and blamed their run ids for it. Treat a pre-2026-07-12 identical on any scope other than findings as meaningless.

Canonicalization Contract

For every scope, Tester:

• Fetches run A and run B through public REST.
• Extracts the returned artefact array. If a service returns an envelope, known array members such as items, packs, findings, events, linksets, or data are used.
• Removes the volatile fields listed below.
• Sorts rows by the scope sort key.
• Sorts JSON object keys recursively.
• Serializes canonical JSON as UTF-8 without BOM and without insignificant whitespace.
• Normalizes floating-point numbers with G17.
• Hashes the canonical array bytes with SHA-256.

The “run-id filtering” column below is verified against the upstream route tables (2026-07-12), not assumed. A scope marked NO is never fetched and never hashed — it returns indeterminate.

The stripped fields are intentionally limited to volatile wall-clock, per-request, export, cache, and transport identifiers. Preserved fields include content hashes, verdicts, policy versions, previous event hashes, payload hashes, linkset hashes, member counts, and algorithm versions.

Negative Path

To prove drift reporting, seed the two runs from fixture inputs that differ in one non-stripped field (for example a different policyVersion or artifactId on a finding), then compare. Use a measurable scope: mutating an indeterminate scope’s data proves nothing, because the oracle never reads it.

Do not demonstrate drift by UPDATE-ing rows in findings.ledger_events directly. The ledger is append-only and hash-chained (event_hash / previous_hash / merkle_leaf_hash, 001_v1_findings_ledger_baseline.sql:80-105): an in-place edit corrupts the chain and will fail chain-verify, which is a different failure than the drift you are trying to show. Drive drift from the input, not from the store.

Then call POST /api/v1/tester/replay-and-compare for the affected scope. Expected response shape:

{
  "verdict": "drift",
  "perScopeDiffs": [
    {
      "scope": "findings",
      "verdict": "drift",
      "runIdFilteringSupported": true,
      "diff": {
        "onlyInA": [],
        "onlyInB": [],
        "changed": [
          {
            "findingId": "finding-001",
            "fieldNames": ["policyVersion"]
          }
        ]
      }
    }
  ]
}

Error Cases

• Empty or missing scope: 400 {"error":"scope_required"}.
• Unknown scope: 400 {"error":"unsupported_scope","scope":"<bad>"}.
• Missing run in at least one measurable downstream service: 404 {"error":"run_not_found","runId":"<id>","scope":"<scope>"}.
• Downstream timeout, 5xx, non-JSON response, or client failure: 502 {"error":"upstream_unavailable","scope":"<scope>","service":"<service>"}.

Note that an unmeasurable scope is not an error: it is a successful 200 whose per-scope verdict is indeterminate. It used to surface as 404 run_not_found / 502 upstream_unavailable, which misattributed a missing upstream feature to the operator’s run ids.

Cross-Reference

This runbook closes the Tester replay-and-compare product gap called out by TOPO-103 in docs/qa/feature-checks/plans/2026-05-18-operator-trust/TIER_5_DETERMINISM_TIME.md. That plan file is owned by the Tier 5 verification stream; this runbook is the operator-facing recipe used by Sprint 20260518_058 TASK-058-2.