Score Proofs Operations Runbook

Version: 1.1.0
Sprint: 3500.0004.0004
Last Updated: 2026-05-31 (doc↔code reconciliation: metrics, escalation CLI, links)

Audience: Scanner and Platform on-call, plus auditors and CI/CD owners who verify Stella Ops scoring decisions.

This runbook covers operational procedures for Score Proofs — the cryptographically verifiable audit trail behind every vulnerability-scoring decision — including score replay, proof verification, proof-bundle management, and troubleshooting.

Table of Contents

1. Overview
2. Score Replay Operations
3. Proof Verification Operations
4. Proof Bundle Management
5. Troubleshooting
6. Monitoring & Alerting
7. Escalation Procedures

1. Overview

What are Score Proofs?

Score Proofs provide cryptographically verifiable audit trails for vulnerability scoring decisions. Each proof:

• Records inputs: SBOM, feed snapshots, VEX data, policy hashes
• Traces computation: Every scoring rule application
• Signs results: DSSE envelopes with configurable trust anchors
• Enables replay: Same inputs → same outputs (deterministic)

Key Components

Prerequisites

• Access to Scanner WebService API (/api/v1 base path).
• OAuth scopes. The endpoints are guarded by ASP.NET authorization policies (names in StellaOps.Scanner.WebService/Security/ScannerPolicies.cs); each policy accepts one of several scope claims (registered in Program.cs via AddStellaOpsAnyScopePolicy). The service-specific scope strings live in ScannerAuthorityScopes.cs, and the read/write policies additionally accept the canonical colon-form scopes from StellaOps.Auth.Abstractions/StellaOpsScopes.cs (scanner:read, scanner:write):
  • scanner.scans.read policy — read manifests, proof bundles, score bundle/history. Accepts scope scanner.scans.read or scanner:read (StellaOpsScopes.ScannerRead).
  • scanner.scans.write policy — invoke score replay and verify. Accepts scope scanner.scans.write or scanner:write (StellaOpsScopes.ScannerWrite).
  • scanner.triage.write policy — generate triage proof bundles (POST /api/v1/triage/proof-bundle). There is no scanner.triage.write scope; this policy is satisfied by the same scopes as scanner.scans.write — i.e. scanner.scans.write or scanner:write (see Program.cs AddStellaOpsAnyScopePolicy(ScannerPolicies.TriageWrite, ScannerAuthorityScopes.ScansWrite, StellaOpsScopes.ScannerWrite)).

  There is no scanner.proofs scope, and none of the dotted scanner.scans.* strings appear in the canonical catalog (StellaOpsScopes.cs) — they are defined in ScannerAuthorityScopes.cs. The catalog’s canonical Scanner scopes are colon-form (scanner:read, scanner:scan, scanner:export, scanner:write); the colon-form read/write scopes satisfy the policies above.

• CLI access with stella configured.
• Trust anchor public keys (for offline attestation-bundle verification).

Feature flag: the score replay/verify/bundle/history endpoints are only mapped when ScoreReplay.Enabled is true in ScannerWebServiceOptions (default: false — see Program.cs if (resolvedOptions.ScoreReplay.Enabled)). If these endpoints return 404, confirm the flag is enabled in the Scanner WebService configuration. The scan manifest and /proofs listing endpoints (below) are always available.

2. Score Replay Operations

2.1 When to Replay Scores

Score replay is needed when:

• Feed updates: New advisories from Concelier
• VEX updates: New VEX statements from Excititor
• Policy changes: Updated scoring policy rules
• Audit requests: Need to verify historical scores
• Investigation: Analyze why a score changed

2.2 Manual Score Replay (API)

Route note: the Scanner WebService base path is /api/v1 (no /scanner segment). $SCAN_ID for the manifest and /proofs endpoints must be a GUID (the handlers reject non-GUID ids with 404 — see ManifestEndpoints.HandleGetManifestAsync).

# Get the scan manifest (input hashes for reproducibility)
# GET /api/v1/scans/{scanId}/manifest — scope: scanner.scans.read
curl -s "https://scanner.example.com/api/v1/scans/$SCAN_ID/manifest" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

# Request the DSSE-signed manifest via content negotiation
curl -s "https://scanner.example.com/api/v1/scans/$SCAN_ID/manifest" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/dsse+json" | jq '.envelope'

# Replay the score deterministically from frozen manifest inputs
# POST /api/v1/scans/{scanId}/score/replay — scope: scanner.scans.write
# Body is optional; both fields may be omitted to replay against the stored manifest.
curl -X POST "https://scanner.example.com/api/v1/scans/$SCAN_ID/score/replay" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}' | jq '.rootHash'

# Replay against a specific manifest hash, with a freeze timestamp
curl -X POST "https://scanner.example.com/api/v1/scans/$SCAN_ID/score/replay" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "manifestHash": "sha256:specific-manifest-hash...",
    "freezeTimestamp": "2026-01-15T03:00:00Z"
  }'

The replay request body is ScoreReplayRequest and accepts only the optional manifestHash and freezeTimestamp fields (see ScoreReplayEndpoints.cs). There is no overrides object and no concelierSnapshotHash field; replay re-uses the frozen manifest inputs rather than substituting feed snapshots. The response is ScoreReplayResponse (score, rootHash, bundleUri, manifestHash, manifestDigest, canonicalInputHash, canonicalInputPayload, seedHex, factors[], verificationStatus, replayedAt, deterministic).

A backward-compatible alias POST /api/v1/score/{scanId}/replay is also mapped (see the legacy group in ScoreReplayEndpoints.cs) and is retained while clients migrate.

2.3 Manual Score Replay (CLI)

The wired-up stella score command group is implemented in ScoreReplayCommandGroup.cs (see CommandFactory line 157). The replay verb takes a scan id via --scan/-s (not --scan-id):

# Replay a score for a scan (frozen manifest inputs → deterministic result)
stella score replay --scan $SCAN_ID

# Replay against a specific manifest hash
stella score replay --scan $SCAN_ID --manifest-hash sha256:specific-manifest-hash...

# Replay with a freeze timestamp (ISO 8601) for deterministic replay
stella score replay --scan $SCAN_ID --freeze 2026-01-15T03:00:00Z

# JSON output
stella score replay --scan $SCAN_ID --output json

Available stella score subcommands (per ScoreReplayCommandGroup.BuildScoreCommand): replay, bundle, verify, explain. Flags such as --feed-snapshot, --diff, --offline, --bundle, --use-original-snapshots, --skip-unchanged, and --quiet are not implemented on these commands.

The stella score explain <digest> subcommand (also part of ScoreReplayCommandGroup) renders a score-factor breakdown for an image digest (not a scan id) by calling the Signals endpoint GET /api/v1/score/explain/{digest} through the authenticated Backend Router. Its options are --format/-f (table (default) | json | markdown), --server (overrides STELLAOPS_BACKEND_URL), and the global --verbose. It is a Signals read path, distinct from the Scanner-side replay/bundle/verify verbs above.

NOT IMPLEMENTED: there is no CLI offline/air-gap mode for score replay. Offline verification is performed against an attestation bundle file with stella proof verify (see §3.2).

2.4 Batch Score Replay

NOT IMPLEMENTED (roadmap). There is no POST /api/v1/scanner/batch/replay endpoint in the Scanner WebService, and stella scan list / a --quiet replay flag do not exist. For bulk replay today, iterate over scan ids and call the per-scan replay endpoint or stella score replay --scan <id> in a loop:

# Driver loop over a list of GUID scan ids you already have
for SCAN_ID in "$@"; do
  echo "Replaying $SCAN_ID..."
  stella score replay --scan "$SCAN_ID" --output json
done

A server-side batch/parallel replay API and a stella scan list enumerator are forward-looking and would need a sprint before this section can document them.

2.5 Automatic Feed-Change Rescore Job

Automatic rescoring is implemented as FeedChangeRescoreJob, a BackgroundService hosted inside the Scanner WebService (not a Scheduler cron job). It polls for feed-snapshot changes and rescroes affected scans via IScoreReplayService. There is no concelier-snapshot-published Scheduler trigger or batch-replay action type, and no nightly-score-replay job.

Configuration (FeedChangeRescoreOptions, with defaults):

# Scanner WebService configuration — FeedChangeRescoreOptions
FeedChangeRescore:
  Enabled: true                # default true
  CheckInterval: "00:15:00"    # poll every 15 minutes (TimeSpan)
  MaxScansPerCycle: 100        # cap rescored scans per cycle
  ScanAgeLimit: "7.00:00:00"   # only consider scans newer than 7 days
  RescoreConcurrency: 4        # parallel rescore operations

The job is interval-driven (default every 15 minutes), not a 3 AM nightly cron. To monitor it, watch the Scanner WebService logs and the StellaOps.Scanner.FeedChangeRescore activity source rather than a Scheduler job-status CLI (those verbs do not exist for this flow).

2.6 Score Replay History

The score group also exposes a read-only history endpoint that returns prior deterministic replays for a scan (useful for explainability timelines and confirming a rescore happened).

# GET /api/v1/scans/{scanId}/score/history — scope: scanner.scans.read
# (legacy alias: GET /api/v1/score/{scanId}/history)
curl -s "https://scanner.example.com/api/v1/scans/$SCAN_ID/score/history" \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | {rootHash, replayedAt, score, canonicalInputHash, manifestDigest}'

Each item is a ScoreHistoryResponseItem (rootHash, replayedAt, score, canonicalInputHash, manifestDigest, factors[]) — see ScoreReplayEndpoints.cs HandleGetHistoryAsync. There is no CLI verb for history; stella score exposes only replay, bundle, verify, and explain.

3. Proof Verification Operations

3.1 Online Verification (score bundle)

Online verification recomputes the proof ledger root hash for a scan and compares it to an expected value. It is done against the score-verify endpoint, not a standalone /proofs/verify endpoint (which does not exist).

# POST /api/v1/scans/{scanId}/score/verify — scope: scanner.scans.write
# expectedRootHash is required; the other fields are optional.
curl -X POST "https://scanner.example.com/api/v1/scans/$SCAN_ID/score/verify" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "expectedRootHash": "sha256:proof123...",
    "bundleUri": "optional-bundle-uri",
    "expectedCanonicalInputHash": "sha256:optional...",
    "canonicalInputPayload": "optional-canonical-json"
  }'

# Verify via CLI (ScoreReplayCommandGroup): requires --scan and --root-hash
stella score verify --scan $SCAN_ID --root-hash sha256:proof123...

The verify response is ScoreVerifyResponse and reports valid, computedRootHash, expectedRootHash, manifestValid, ledgerValid, canonicalInputHashValid, expectedCanonicalInputHash, canonicalInputHash, verifiedAtUtc, and errorMessage (see ScoreReplayEndpoints.cs). The CLI score verify exits non-zero when valid is false.

NOT IMPLEMENTED: there is no Rekor/transparency-log check or anchorIds parameter on the score-verify endpoint. Score verification is root-hash + signature/ledger based. Rekor-backed verification applies to the attestation-bundle flow in §3.2.

3.2 Offline Verification (Air-Gap) — attestation bundle

stella proof verify (implemented in ProofCommandGroup.cs) verifies an attestation bundle file (.tar.gz), not the score-replay ZIP. It supports an --offline switch that skips transparency-log (Rekor) verification.

# 1. Obtain the proof/attestation bundle on a connected system
#    (e.g. via the export/attestor flow that produced it).

# 2. Transfer to the air-gapped system (USB, etc.)

# 3. Verify offline (skips Rekor). --bundle/-b is the path to the .tar.gz.
stella proof verify --bundle proof-bundle.tar.gz --offline

# 4. Online verification (includes transparency-log check)
stella proof verify --bundle proof-bundle.tar.gz

# JSON output for CI parsing
stella proof verify --bundle proof-bundle.tar.gz --offline --output json

The only flags on stella proof verify are --bundle/-b (required), --offline, --output (text|json), and the global --verbose. There are no --bundle-id, --trust-anchor, --public-key, --skip-rekor, or --check flags — those are not implemented for this command. Offline mode is expressed by --offline, which sets VerifyTransparency: false.

The proof command group exposes only verify and spine show <bundle-id> (stella proof spine show currently prints a not-yet-implemented placeholder).

3.2a CI/CD Gate Verification Quick Reference

Concise commands for CI/CD pipeline verification gates.

FLAGGED — not verified against this repository’s CLI. The ProofCommandGroup in src/Cli implements stella proof verify --bundle <path.tar.gz> [--offline] [--output] and exposes no --image, --check-rekor, --fail-on-missing, or --ledger-path options, and there is no evidence-pack command in this group. The snippets below were authored under SPRINT_20260112_004_DOC_cicd_gate_verification and may describe a different/forward-looking CLI surface. Confirm against the live CLI before relying on them in a gate; the working invocations are in §3.1 (stella score verify) and §3.2 (stella proof verify --bundle).

Online (transparency-log backed) — UNVERIFIED:

stellaops proof verify --image $IMAGE --check-rekor --fail-on-missing

Offline (local ledger) — UNVERIFIED:

stellaops proof verify --image $IMAGE --offline --ledger-path /var/lib/stellaops/ledger

Evidence pack verification — UNVERIFIED:

stellaops evidence-pack verify --bundle $PACK_PATH --check-signatures --check-merkle

See also: CI/CD Gate Flow - DSSE Witness Verification | Proof Verification Runbook

3.3 Verification Checks

For the attestation bundle flow (stella proof verify), the checks reported are (per ProofCommandGroup.BuildVerificationChecks / PrintTextResult):

For the score bundle flow (POST /api/v1/scans/{scanId}/score/verify), the response fields are manifestValid (manifest DSSE signature), ledgerValid (proof ledger integrity), canonicalInputHashValid (canonical input hash match), and valid (overall), with computedRootHash compared against expectedRootHash.

The earlier “ID Recomputed”, “Merkle Path Valid”, and “Timestamp Valid” rows do not map to fields/checks emitted by either command and have been removed.

3.4 Failed Verification Troubleshooting

# Detailed report (global --verbose flag is supported)
stella proof verify --bundle proof-bundle.tar.gz --verbose

# JSON output exposes the per-check results (file_integrity, dsse_signature,
# transparency_log) for programmatic triage
stella proof verify --bundle proof-bundle.tar.gz --output json

NOT IMPLEMENTED: stella proof verify --check <name>, stella proof inspect, and a --bundle-id lookup do not exist. The attestation-bundle verifier always runs all applicable checks; use --output json to inspect individual check statuses.

4. Proof Bundle Management

4.1 List and Locate Proof Bundles

There is no stella proof download command. Proof bundles are enumerated and located via the manifest/proofs API (all require scanner.scans.read), and the score bundle’s storage URI is surfaced by stella score bundle:

# List all proof bundles for a scan
# GET /api/v1/scans/{scanId}/proofs
curl -s "https://scanner.example.com/api/v1/scans/$SCAN_ID/proofs" \
  -H "Authorization: Bearer $TOKEN" | jq '.items[] | {rootHash, bundleType, bundleHash, createdAt}'

# Get a specific proof bundle's metadata by root hash
# GET /api/v1/scans/{scanId}/proofs/{rootHash}
curl -s "https://scanner.example.com/api/v1/scans/$SCAN_ID/proofs/$ROOT_HASH" \
  -H "Authorization: Bearer $TOKEN" | jq '.'

# Resolve the score bundle storage URI (and check the manifest DSSE state)
# GET /api/v1/scans/{scanId}/score/bundle — scope: scanner.scans.read
stella score bundle --scan $SCAN_ID --output json

The bundleUri returned by score bundle/score replay is a server-side storage path (default base /var/lib/stellaops/proofs, see ProofBundleWriterOptions.StorageBasePath). The /proofs/{rootHash} API returns bundle metadata (ProofBundleResponse), not the ZIP bytes; there is no streaming download endpoint for the ZIP in this service.

4.2 Bundle Contents

The proof bundle ZIP is written by ProofBundleWriter.CreateZipBundleAsync with exactly four entries:

# List bundle contents
unzip -l proof-bundle.zip

# Actual contents (ProofBundleWriter):
#   manifest.json        - Scan manifest (canonical JSON)
#   manifest.dsse.json   - Signed manifest with DSSE envelope
#   score_proof.json     - Proof ledger payload { nodes[], rootHash, createdAtUtc }
#   meta.json            - Bundle metadata { rootHash, createdAtUtc, version }

# Extract and inspect
unzip proof-bundle.zip -d ./proof-contents/
cat ./proof-contents/manifest.json | jq .
cat ./proof-contents/score_proof.json | jq '.nodes | length'

The writer does not emit a separate proof_root.dsse.json entry (the root-hash DSSE referenced in the class doc-comment is not written). The root hash is carried in both score_proof.json and meta.json.

4.3 Proof Retention

DRAFT / ROADMAP. A tiered Hot/Warm/Cold retention model and the stella proof status / proof retrieve / proof retrieve-status commands are not implemented in this repository. ProofBundleRow/ProofBundleResponse carry an expiresAt timestamp (ExpiresAt), but there is no cold-storage retrieval workflow or CLI surface. The table below is an illustrative target, not current behavior:

4.4 Export for Audit

NOT IMPLEMENTED. stella proof export and stella proof export-batch do not exist in the proof command group (which provides only verify and spine show). For audit today, collect proof bundles via the /api/v1/scans/{scanId}/proofs listing (§4.1) and the attestation-bundle verification flow (§3.2). A first-class audit-export verb is forward-looking and needs a sprint.

5. Troubleshooting

5.1 Score Mismatch After Replay

Symptom: Replayed score differs from original.

Diagnosis:

# Compare manifests
stella score diff --scan-id $SCAN_ID --original --replayed

# Check for feed changes
stella score manifest --scan-id $SCAN_ID | jq '.concelierSnapshotHash'

# Compare input hashes
stella score inputs --scan-id $SCAN_ID --hash

Common causes:

1. Feed snapshot changed: Original used different advisory data
2. Policy updated: Scoring rules changed between runs
3. VEX statements added: New VEX data affects scores
4. Non-deterministic seed: Check if deterministic: true in manifest

Resolution:

# Pin the replay to the original manifest hash for an exact comparison
# (read the original hash from GET /api/v1/scans/{scanId}/manifest first).
stella score replay --scan $SCAN_ID --manifest-hash $ORIGINAL_MANIFEST_HASH

NOT IMPLEMENTED: there is no --use-original-snapshots flag and no score diff / score inputs / score manifest CLI verbs. Compare manifests by fetching GET /api/v1/scans/{scanId}/manifest (returns manifestHash, sbomHash, rulesHash, feedHash, policyHash) for each scan and diffing the hashes yourself.

5.2 Proof Verification Failed

Symptom: Verification returns valid: false (score verify) or a non-zero exit / status: fail (attestation-bundle proof verify).

Diagnosis:

# Attestation bundle: detailed, machine-readable check results
stella proof verify --bundle proof-bundle.tar.gz --output json | jq '.checks'

# Score bundle: inspect which sub-check failed
stella score verify --scan $SCAN_ID --root-hash $ROOT_HASH --output json \
  | jq '{valid, manifestValid, ledgerValid, canonicalInputHashValid, computedRootHash, expectedRootHash}'

Resolution by failing check:

The “ID recomputation failed” / “Merkle path invalid” error strings in the prior revision are not produced by either verifier and have been replaced with the actual check names.

5.3 Missing Proof Bundle

Symptom: Proof bundle not found (404 from /proofs or score bundle).

Diagnosis:

# Confirm the scan exists and check its status
# GET /api/v1/scans/{scanId}
curl -s "https://scanner.example.com/api/v1/scans/$SCAN_ID" \
  -H "Authorization: Bearer $TOKEN" | jq '.status'

# List proof bundles that were actually generated
curl -s "https://scanner.example.com/api/v1/scans/$SCAN_ID/proofs" \
  -H "Authorization: Bearer $TOKEN" | jq '.items'

Common causes:

1. Scan still in progress: the proof is generated after completion.
2. Proof generation failed: check Scanner worker logs.
3. Score replay disabled: the /score/* endpoints are 404 when ScoreReplay.Enabled is false (§1 Prerequisites).
4. Non-GUID scan id: the manifest/proofs handlers require a GUID and return 404 otherwise.

NOT IMPLEMENTED: stella scan status, stella proof status, and stella proof list. Use the HTTP endpoints above.

5.4 Replay Performance Issues

Symptom: Replay taking too long.

Diagnosis:

• Inspect the Scanner WebService telemetry (see §6 for the actual emitted metric names — there is no score_replay_duration_seconds metric; the live rescore counters/histogram come from the StellaOps.Scanner.FeedChangeRescore meter) and host-level resource metrics for the scanner-web container.

NOT IMPLEMENTED: stella scheduler queue status, stella scanner workers status|metrics, and stella scheduler job update are not part of the score-proofs CLI surface, and there is no batch/queue replay subsystem (see §2.4/§2.5). The --skip-unchanged flag on score replay does not exist. Performance tuning is therefore via service configuration and host resources, not a replay queue.

6. Monitoring & Alerting

6.1 Key Metrics

FLAGGED — metric names corrected to source. The only score-proof / replay metrics actually emitted by this repository are the four instruments on the StellaOps.Scanner.FeedChangeRescore meter (FeedChangeRescoreMetrics), driven by the in-process feed-change rescore loop (§2.5). The score_replay_duration_seconds, proof_verification_success_rate, proof_bundle_size_bytes, replay_queue_depth, and proof_generation_failures names used in the prior revision (and echoed in §6.3 / §7) are not implemented — there is no replay queue subsystem and no per-replay duration or verification-success metric. Use the real instruments below until those are added.

Actually emitted (meter StellaOps.Scanner.FeedChangeRescore, v1.0.0):

NOT IMPLEMENTED (roadmap thresholds): a per-replay latency histogram, a verification success-rate gauge, a proof-bundle-size gauge, and a replay-queue-depth gauge. The threshold table below is an illustrative target, not current behavior, and references metrics that do not yet exist:

6.2 Grafana Dashboard

Dashboard: Score Proofs Operations
Panels:
- Replay throughput (replays/minute)
- Replay latency (p50, p95, p99)
- Verification success rate
- Proof bundle storage usage
- Queue depth over time

6.3 Alerting Rules

FLAGGED — illustrative only. The score_replay_duration_seconds, proof_verification_failures_total, and replay_queue_depth series referenced below are not emitted by this repository (see §6.1). These rules will not fire against the current build; they are retained as a target shape for when those metrics are introduced. Alerting that works today must be built on the stellaops.scanner.* instruments in §6.1.

# Prometheus alerting rules (ROADMAP — series not yet emitted; see §6.1)
groups:
  - name: score-proofs
    rules:
      - alert: ReplayLatencyHigh
        expr: histogram_quantile(0.95, score_replay_duration_seconds) > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Score replay latency is high"
          
      - alert: ProofVerificationFailures
        expr: increase(proof_verification_failures_total[1h]) > 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Multiple proof verification failures detected"
          
      - alert: ReplayQueueBacklog
        expr: replay_queue_depth > 1000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Score replay queue backlog is growing"

7. Escalation Procedures

7.1 Escalation Matrix

7.2 P1 Response Procedure

FLAGGED — the CLI invocations in the prior revision did not match the implemented stella surface. Corrections, verified against src/Cli:

• There is no top-level stella health check --service <svc> command. Health verbs are scoped per area (e.g. stella admin health, stella integrations health, stella agent health) — none take a --service selector. Scanner WebService health is surfaced via its /healthz//readyz endpoints and telemetry (§6.1), not a health check verb.
• The release command group is release(singular), not releases, and exposes create, promote, rollback, list, show, hooks, verify, ci, deploy, gates, status (ReleaseCommandGroup.BuildReleaseCommand). There is no history subcommand and no --service/--to flags; rollback operates on environment/version (--env, release version), not a per-service --to previous.
• The anchor command group exposes list, show, create, revoke-key (AnchorCommandGroup). There is no restore subcommand and no --anchor-id/--revision flags; moreover the implemented anchor verbs are currently TODO placeholders that print “(implementation pending)”.

1. Acknowledge alert in your on-call tooling.
2. Triage:

   # Scanner / Attestor liveness+readiness via the service health endpoints
   curl -fsS https://scanner.example.com/healthz && curl -fsS https://scanner.example.com/readyz
   
   # Inspect recent releases via the orchestrator API (the former `stella release
   # list/show` commands emitted fabricated sample data and were removed 2026-07-03;
   # `stella release` now carries only approve/reject/promote/deploy/rollback)
   curl -fsS -H "Authorization: Bearer $TOKEN" \
     https://<gateway>/api/v1/release-orchestrator/releases
   curl -fsS -H "Authorization: Bearer $TOKEN" \
     https://<gateway>/api/v1/release-orchestrator/releases/<releaseId>
   

3. Mitigate:

   # If a recent deployment caused the incident, roll back via the release command group
   # (operates on environment + release version, not a per-service `--to previous`).
   stella release rollback --env prod   # see `stella release rollback --help` for required args
   

   NOT IMPLEMENTED: a one-shot “restore previous trust anchor” verb. The anchor group has list/show/create/revoke-key only, and those are placeholders. Key-rotation recovery is a manual/Signer-side procedure today, not a CLI command.

4. Communicate: notify stakeholders and update the incident channel/status page.
5. Resolve: Fix root cause, verify fix.
6. Postmortem: Document incident afterward per your team’s policy.

7.3 Contact Information

Related Documentation

• Score Replay Operations Runbook
• Score Proofs API Reference
• Proof Chain Architecture
• CLI Reference
• Air-Gap Operations
• Score Proofs / Reachability Air-Gap Runbook
• Proof Verification Runbook

Last Updated: 2026-05-31
Version: 1.1.0
Sprint: 3500.0004.0004