Score Replay Operations Runbook

Version: 1.1.1
Sprint: 3500.0004.0004
Last Updated: 2026-05-31

Audience: Scanner and Signals on-call, plus auditors verifying that a Stella Ops vulnerability score is reproducible.

This runbook covers operational procedures for Score Replay, including deterministic score-computation verification, proof-bundle validation, and troubleshooting replay discrepancies. For the broader proof-bundle lifecycle, monitoring, and escalation, see the Score Proofs Operations Runbook.

Scope note. This runbook documents the Scanner score-replay surface (stella score … CLI verbs and the Scanner WebService /score and /scans/{scanId}/score endpoints), which recompute a vulnerability score for an existing scan from its frozen manifest and emit a signed proof bundle. It is distinct from the platform-wide deterministic Replay service (src/Replay/, replay:read/replay:write scopes) and from the Findings-ledger replay harness (src/Findings/.../LedgerReplayHarness), which are out of scope here.

Table of Contents

1. Overview
2. Score Replay Operations
3. Determinism Verification
4. Proof Bundle Management
5. Troubleshooting
6. Monitoring & Alerting
7. Escalation Procedures

1. Overview

What is Score Replay?

Score Replay is the ability to re-execute a vulnerability score computation using the exact same inputs (SBOM, rules, policies, feeds) that were used in the original scan. This provides:

• Auditability: Prove that a score was computed correctly
• Determinism verification: Confirm that identical inputs produce identical outputs
• Compliance evidence: Generate proof bundles for regulatory requirements
• Dispute resolution: Verify contested scan results

Key Concepts

Architecture Components

2. Score Replay Operations

2.1 Triggering a Score Replay

Via CLI

The stella score replay verb is implemented in ScoreReplayCommandGroup (src/Cli/StellaOps.Cli/Commands/ScoreReplayCommandGroup.cs). The scan is selected with --scan/-s (required); --output/-o accepts text (default) or json.

# Basic replay
stella score replay --scan <scan-id>

# Replay with specific manifest
stella score replay --scan <scan-id> --manifest-hash sha256:abc123...

# Replay with frozen timestamp (for determinism testing; ISO 8601)
stella score replay --scan <scan-id> --freeze 2025-01-15T00:00:00Z

# Output as JSON
stella score replay --scan <scan-id> --output json

The CLI posts to /api/v1/scanner/score/{scanId}/replay against the ScannerApi client (the gateway routes /api/v1/scanner/* to the Scanner service, where it resolves to the /score/{scanId}/replay legacy alias).

Via API

The endpoints are registered in ScoreReplayEndpoints.cs under the Scanner API base path (/api/v1). Two route shapes exist:

• Canonical — POST /api/v1/scans/{scanId}/score/replay
• Legacy alias (retained while clients migrate) — POST /api/v1/score/{scanId}/replay

replay and verify require the ScansWrite policy; bundle and history require the ScansRead policy (registered in ScoreReplayEndpoints.MapScoreReplayEndpoints). These are ASP.NET policy names, not raw scope strings. Each policy is an any-of scope check (AddStellaOpsAnyScopePolicy, requireAllScopes: false, see Program.cs):

• ScansWrite is satisfied by either the Scanner-local scope scanner.scans.write or the canonical catalog scope scanner:write (StellaOpsScopes.ScannerWrite).
• ScansRead is satisfied by either scanner.scans.read or the canonical scanner:read (StellaOpsScopes.ScannerRead).

The dotted scanner.scans.read / scanner.scans.write forms are Scanner-WebService-local scopes declared in ScannerAuthorityScopes.cs; they are not in the canonical scope catalog (StellaOpsScopes.cs, which carries only scanner:read, scanner:scan, scanner:export, scanner:write). When the service runs with Authority disabled (anonymous mode), all Scanner policies allow-all.

# Canonical route
curl -X POST "https://stella-ops.local/api/v1/scans/scan-123/score/replay" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "manifestHash": "sha256:abc123...",
    "freezeTimestamp": "2025-01-15T00:00:00Z"
  }'

Both request fields are optional. When freezeTimestamp is omitted the replay falls back to the manifest’s createdAtUtc.

Expected Response

The response body is ScoreReplayResponse. Note that score is normalised to the 0.0–1.0 range (not a 0–10 CVSS value), and there is no scanId echo field. Example (camelCase JSON):

{
  "score": 0.752,
  "rootHash": "sha256:def456...",
  "bundleUri": "/var/lib/stellaops/proofs/scan-123_def4567890abcdef.zip",
  "manifestHash": "sha256:abc123...",
  "manifestDigest": "sha256:...",
  "canonicalInputHash": "sha256:...",
  "canonicalInputPayload": "{...}",
  "seedHex": "...",
  "factors": [
    { "name": "reachability", "weight": 0.30, "raw": 1.0, "weighted": 0.30, "source": "..." }
  ],
  "verificationStatus": "verified",
  "replayedAt": "2025-01-16T10:30:00Z",
  "deterministic": true
}

deterministic reflects the stored manifest’s Deterministic flag — it is not a live two-run comparison. bundleUri is a server-side filesystem path to the ZIP bundle, not a downloadable HTTP URL.

2.2 Retrieving Proof Bundle Metadata

Via CLI

stella score bundle returns bundle metadata (scan id, root hash, bundle URI, DSSE validity, created-at) as text or JSON. It does not download the ZIP archive itself — --output/-o only selects text (default) or json.

# Get bundle metadata for a scan
stella score bundle --scan <scan-id>

# As JSON
stella score bundle --scan <scan-id> --output json

Via API

# Canonical:   GET /api/v1/scans/{scanId}/score/bundle
# Legacy alias: GET /api/v1/score/{scanId}/bundle
# Optional query: ?rootHash=sha256:...   (select a specific bundle)
curl "https://stella-ops.local/api/v1/scans/scan-123/score/bundle" \
  -H "Authorization: Bearer $TOKEN"

The JSON body is ScoreBundleResponse (scanId, rootHash, bundleUri, manifestDsseValid, createdAt).

2.3 Verifying Score Integrity

Via CLI

stella score verify requires both --scan/-s and --root-hash/-r, and accepts an optional --bundle-uri/-b to pin a specific bundle. Exit codes: 0 when the API returns 200 with valid:true; 2 when the API returns 200 with valid:false (signature/ledger/hash mismatch); and 1 on any non-2xx response (including 404 bundle-not-found, 400 bad request, or a network/transport error) or an empty response body.

# Verify against expected root hash (both flags required)
stella score verify --scan <scan-id> --root-hash sha256:def456...

# Pin a specific bundle artifact
stella score verify --scan <scan-id> --root-hash sha256:def456... \
  --bundle-uri /var/lib/stellaops/proofs/scan-123_def4567890abcdef.zip

# Verify a saved attestation bundle (.zip) via the proof verifier
stella proof verify --bundle bundle.zip

The CLI score verify request carries only expectedRootHash and bundleUri; the optional expectedCanonicalInputHash / canonicalInputPayload verify fields are API-only (no CLI flag wires them).

Via API

# Canonical:   POST /api/v1/scans/{scanId}/score/verify
# Legacy alias: POST /api/v1/score/{scanId}/verify
curl -X POST "https://stella-ops.local/api/v1/scans/scan-123/score/verify" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"expectedRootHash": "sha256:def456..."}'

expectedRootHash is mandatory (a blank value returns 400). Optional fields: bundleUri, expectedCanonicalInputHash, canonicalInputPayload. The ScoreVerifyResponse reports valid, computedRootHash, manifestValid, ledgerValid, and canonicalInputHashValid.

2.4 Score Replay History

A read-only history endpoint returns the deterministic replay records observed for a scan, used to drive explainability timelines:

# Canonical:   GET /api/v1/scans/{scanId}/score/history
# Legacy alias: GET /api/v1/score/{scanId}/history
curl "https://stella-ops.local/api/v1/scans/scan-123/score/history" \
  -H "Authorization: Bearer $TOKEN"

Implementation note. ScoreReplayService keeps history in an in-memory map keyed by scan id (_historyByScan). It is populated by replays performed in the current process and is not read back from the score_replay_history table. History therefore resets on service restart and is not shared across Scanner instances.

3. Determinism Verification

3.1 What Affects Determinism?

Score computation is deterministic when:

3.2 Running Determinism Checks

# Run replay twice and compare
REPLAY1=$(stella score replay --scan $SCAN_ID --output json)
REPLAY2=$(stella score replay --scan $SCAN_ID --output json)

# Extract root hashes
HASH1=$(echo $REPLAY1 | jq -r '.rootHash')
HASH2=$(echo $REPLAY2 | jq -r '.rootHash')

# Compare
if [ "$HASH1" = "$HASH2" ]; then
  echo "✓ Determinism verified: $HASH1"
else
  echo "✗ Non-deterministic! $HASH1 != $HASH2"
  exit 1
fi

3.3 Common Determinism Issues

3.4 Feed Freeze for Reproducibility

# Replay with feed state frozen to the original scan time.
# (The replay also frozen-derives this from the manifest's createdAtUtc when
#  --freeze is omitted.)
stella score replay --scan $SCAN_ID \
  --freeze $(stella scan show $SCAN_ID --output json | jq -r '.scannedAt')

The exact JSON field exposed by stella scan show for the original scan time was not verified against the current Scanner contract; confirm the field name (.scannedAt here is illustrative) before scripting against it.

4. Proof Bundle Management

4.1 Bundle Contents

A score proof bundle is a ZIP archive ({scanId}_{rootHashPrefix16}.zip, or {scanId}.zip when content-addressing is disabled), written by ProofBundleWriter under its configured StorageBasePath (default /var/lib/stellaops/proofs). It contains exactly these entries:

bundle.zip/
├── manifest.json        # Canonical JSON scan manifest (input hashes + metadata)
├── manifest.dsse.json   # DSSE-signed envelope wrapping the manifest
├── score_proof.json     # ProofLedger nodes array (Merkle leaves)
└── meta.json            # Bundle metadata (root hash, created-at)

The writer’s docstring also lists an optional proof_root.dsse.json (DSSE envelope for the root hash), but the current CreateZipBundleAsync implementation does not emit it. There is no score.json, merkle-proof.json, dsse-envelope.json, or certificate.pem.

4.2 Inspecting Bundles

# Extract and view the manifest
unzip -o bundle.zip
cat manifest.json | jq .

# Inspect the proof ledger (Merkle leaves) and bundle metadata
cat score_proof.json | jq .
cat meta.json | jq .

# Verify an attestation bundle's proof chain (DSSE + transparency)
stella proof verify --bundle bundle.zip --verbose

stella proof spine show <bundle-id> exists but its backend retrieval is a TODO stub (HandleSpineShowAsync prints “Spine display not yet implemented”). Use stella proof verify --bundle <path> for local bundle verification.

4.3 Bundle Retention Policy

The proof_bundle table carries an optional expires_at column (indexed for expiry sweeps), so per-environment TTLs can be applied at write time. The values below are a recommended operator policy, not an automatic behaviour enforced by the current code — there is no built-in retention sweeper verified in src/, so any cleanup must be wired explicitly.

4.4 Archiving Bundles

Not implemented as documented. The stella score group has no bundle-export subcommand, and stella score bundle returns metadata only — it does not write the ZIP to a file. To archive a bundle, copy the artifact at the bundleUri reported by score bundle/score replay (a server-side filesystem path to the .zip) using your standard file-transfer tooling, or use the dedicated bundle-export tooling where available (stella bundle export …, BundleExportCommand).

5. Troubleshooting

5.1 Replay Returns Different Score

Symptoms: Replayed score differs from original scan score.

Diagnostic Steps:

1. Check manifest integrity:

   stella scan show $SCAN_ID --output json | jq '.manifest'
   

2. Verify feed/manifest state:

   # Inspect the manifest hash used by a frozen replay
   stella score replay --scan $SCAN_ID --freeze $ORIGINAL_TIME --output json | jq '.manifestHash'
   

3. Compare manifest input hashes directly in the database (see Appendix A) — sbom_hash, rules_hash, feed_hash, policy_hash on scan_manifest capture every scoring input.

There is no stella rules show --version command; rule/policy versions are pinned via the manifest hashes above rather than a dedicated CLI verb.

Resolution:

• Use --freeze timestamp matching the original scan
• Pin rule/policy versions in the manifest
• Regenerate manifest if inputs changed legitimately

5.2 Proof Verification Fails

Symptoms: stella proof verify returns validation errors.

Diagnostic Steps:

1. Check the DSSE signature / verification output:

   stella proof verify --bundle bundle.zip --verbose
   

2. Re-run the score-bundle verify via API to localise the failing check (manifestValid, ledgerValid, canonicalInputHashValid):

   curl -X POST "https://stella-ops.local/api/v1/scans/$SCAN_ID/score/verify" \
     -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
     -d '{"expectedRootHash":"sha256:..."}' | jq .
   

3. Confirm the bundle is intact — a score proof ZIP must contain manifest.json, manifest.dsse.json, score_proof.json, and meta.json (the reader throws if any are missing).

Failure signals (the API surfaces these as errorMessage strings rather than fixed error codes):

5.3 Replay Timeout

Symptoms: Replay request times out or takes too long.

Diagnostic Steps:

1. Check scan size:

   stella scan show $SCAN_ID --output json | jq '.findingsCount'
   

2. Monitor replay progress:

   stella score replay --scan $SCAN_ID --verbose
   

Resolution:

• For large scans (>10k findings), increase timeout
• Check Scanner Worker health
• Consider async replay for very large scans

5.4 Missing Manifest

Symptoms: Manifest not found error on replay.

Diagnostic Steps:

1. Verify scan exists:

   stella scan show $SCAN_ID
   

2. Check the manifest table (note: scan_id is a UUID FK to scans):

   SELECT * FROM scan_manifest WHERE scan_id = '00000000-0000-0000-0000-000000000123';
   

Resolution:

• Manifest may have been purged (check retention policy)
• Restore from backup if available
• Re-run scan if original inputs available

6. Monitoring & Alerting

Status: proposed, not yet emitted. None of the metric names below were found in src/ — there is no dedicated score-replay meter/counter in the Scanner WebService at the time of writing (the only related instrument is FeedChangeRescoreMetrics under the StellaOps.Scanner.FeedChangeRescore ActivitySource, which tracks the feed-driven rescore job, not score replay). Treat the metrics, queries, and alert rules in this section as the target observability contract for score replay; wire the instruments before relying on these dashboards/alerts.

6.1 Key Metrics (proposed)

6.2 Grafana Dashboard Queries

# Replay latency
histogram_quantile(0.99, 
  rate(score_replay_duration_ms_bucket[5m])
)

# Determinism failure rate
rate(score_replay_determinism_failures_total[1h])

# Proof verification success rate
sum(rate(proof_verification_success_total[1h])) /
sum(rate(proof_verification_total[1h]))

6.3 Alert Rules

groups:
  - name: score-replay
    rules:
      - alert: ScoreReplayLatencyHigh
        expr: histogram_quantile(0.99, rate(score_replay_duration_ms_bucket[5m])) > 30000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Score replay latency exceeds 30s at p99

      - alert: DeterminismFailure
        expr: increase(score_replay_determinism_failures_total[1h]) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: Non-deterministic score replay detected

7. Escalation Procedures

7.1 Escalation Matrix

7.2 P1: Determinism Failure Response

1. Immediate Actions (0-15 min):

   • Capture affected scan IDs
   • Preserve original manifest data
   • Check for recent deployments

2. Investigation (15-60 min):

   • Compare input hashes between replays
   • Check feed synchronization status
   • Review rule engine logs

3. Remediation:

   • Roll back if deployment-related
   • Freeze feeds if data drift
   • Hotfix if code bug identified

7.3 Contacts

Appendix A: SQL Queries

Table/column names below match the Scanner storage schema (001_initial_schema.sql, 006_score_replay_tables.sql). The relevant tables are scan_manifest, proof_bundle, and score_replay_history — all unqualified (no scanner. schema prefix). scan_id is a UUID FK to scans.

Check Manifest History

SELECT
  scan_id,
  manifest_hash,
  sbom_hash,
  rules_hash,
  policy_hash,
  feed_hash,
  scanner_version,
  created_at
FROM scan_manifest
WHERE scan_id = '00000000-0000-0000-0000-000000000123'
ORDER BY created_at DESC;

Find Non-Deterministic Replays

Distinct Merkle roots live on proof_bundle (the score_replay_history table has no root_hash column — it records original_manifest_hash / replayed_manifest_hash deltas instead). More than one root hash for a single scan signals non-determinism:

SELECT
  scan_id,
  COUNT(DISTINCT root_hash) AS unique_root_hashes,
  MIN(created_at) AS first_bundle,
  MAX(created_at) AS last_bundle
FROM proof_bundle
GROUP BY scan_id
HAVING COUNT(DISTINCT root_hash) > 1;

Proof Bundle Statistics

proof_bundle stores the archive in bundle_content BYTEA (there is no bundle_size_bytes column); use length(bundle_content) for size:

SELECT
  DATE_TRUNC('day', created_at) AS day,
  COUNT(*) AS bundles_created,
  AVG(length(bundle_content)) AS avg_size_bytes,
  SUM(length(bundle_content)) AS total_size_bytes
FROM proof_bundle
WHERE created_at > NOW() - INTERVAL '30 days'
GROUP BY DATE_TRUNC('day', created_at)
ORDER BY day DESC;

Replay Trigger Audit

SELECT
  scan_id,
  trigger_type,            -- 'feed_update' | 'policy_change' | 'manual' | 'scheduled'
  trigger_reference,
  findings_added,
  findings_removed,
  findings_rescored,
  duration_ms,
  replayed_at
FROM score_replay_history
WHERE scan_id = '00000000-0000-0000-0000-000000000123'
ORDER BY replayed_at DESC;

Appendix B: CLI Quick Reference

# Score Replay Commands (ScoreReplayCommandGroup)
stella score replay --scan <id>                       # Replay score computation
stella score replay --scan <id> --manifest-hash <h>   # Replay against a specific manifest
stella score replay --scan <id> --freeze <iso8601>    # Replay with frozen time
stella score bundle --scan <id>                       # Show proof bundle metadata (no file download)
stella score verify --scan <id> --root-hash <hash>    # Verify score (--scan AND --root-hash required)
stella score verify --scan <id> --root-hash <hash> --bundle-uri <path>  # ...optionally pin a bundle
stella score explain <digest>                         # Explain the risk score breakdown

# Proof Commands (ProofCommandGroup)
stella proof verify --bundle <path.zip>               # Verify an attestation bundle's proof chain
stella proof verify --bundle <path.zip> --offline     # Offline verification (skip transparency log)
stella proof spine show <bundle-id>                   # (backend retrieval is a TODO stub)

# Output Formats for the score replay / bundle / verify verbs
--output text                                # Text output (default)
--output json                                # JSON output
# NB: `score explain` additionally supports `--format table|json|markdown`.

The history and (per-scan) score endpoints are exposed via the API only; the wired score CLI group above (registered at CommandFactory.cs) does not ship a history verb or a bundle-export verb. A separate, richer score group (ScoreCommandGroup: compute/explain/replay/verify/history/compare, targeting the Signals-owned /api/v1/score/{scoreId} surface) exists in the codebase but is not the group bound to the root command.

Revision History