Replay Architecture

Implementation-ready architecture for Replay — the deterministic replay engine of the Stella Ops release control plane. Replay ensures that vulnerability assessments can be reproduced byte-for-byte given the same inputs (SBOM, policy, feeds, toolchain). Audience: engineers building or integrating replay tokens, manifests, feed snapshots, and verification workflows, plus auditors validating deterministic verdict reproducibility.

0) Mission & boundaries

Mission. Enable deterministic reproducibility of vulnerability verdicts. Given identical inputs (SBOM, policy, feeds, toolchain), the system MUST produce identical outputs. Replay provides the infrastructure to capture, store, and verify these deterministic execution chains.

Boundaries.

1) Solution & project layout

Replay code lives across two source roots: the runtime service tree under src/Replay/ and shared replay libraries under src/__Libraries/. Note that there are two StellaOps.Replay.Core libraries (one under each root) plus StellaOps.Replay and StellaOps.Replay.Anonymization.

src/Replay/
 ├─ StellaOps.Replay.WebService/                 # ASP.NET Core minimal-API host
 │   ├─ Program.cs                               # Host, auth policies, token + storage wiring
 │   ├─ VerdictReplayEndpoints.cs                # /v1/replay/verdict endpoints + DTOs
 │   ├─ ReplayBundleProducerEndpoints.cs         # bundle-producer endpoints (MapReplayBundleProducerEndpoints, Program.cs:279)
 │   ├─ ReplayBySubjectEndpoints.cs              # replay-by-subject endpoint (MapReplayBySubjectEndpoint, Program.cs:280)
 │   ├─ PointInTimeQueryEndpoints.cs             # /v1/pit/advisory + /v1/pit/snapshots endpoints
 │   ├─ ReplayFeedSnapshotStores.cs              # PostgresFeedSnapshotIndexStore + SeedFsFeedSnapshotBlobStore
 │   ├─ FeedSnapshotSupport.cs                   # feed-snapshot support helpers
 │   ├─ Services/                                # WebService-local services
 │   ├─ Translations/                            # localized problem-detail strings
 │   ├─ ReplayMigrationAssemblyMarker.cs
 │   └─ Migrations/001_initial_schema.sql        # Embedded `replay` schema migration
 ├─ __Libraries/StellaOps.Replay.Core/           # Runtime replay engine
 │   ├─ DeterminismVerifier.cs                    # VerificationResult + verdict-digest comparison
 │   ├─ ReplayExecutor.cs                         # Replay execution + GenerateReplayId
 │   ├─ ReplayJobQueue.cs                         # Bounded async job queue + GenerateJobId
 │   ├─ InputManifestResolver.cs
 │   ├─ PolicySimulationInputLock.cs              # Input pinning + validator for simulation
 │   ├─ FeedSnapshots/                            # FeedSnapshotService, PointInTimeAdvisoryResolver
 │   ├─ TlptBaselines/                            # TLPT baseline builder + canonical hashing
 │   └─ Providers/                                # Concelier/Policy/VEX upstream ports
 ├─ __Libraries/StellaOps.Replay.Anonymization/   # Trace anonymization helpers
 └─ __Tests/StellaOps.Replay.Core.Tests/          # Unit + integration tests

src/__Libraries/
 ├─ StellaOps.Replay.Core/                        # Replay manifest schema + bundle/export
 │   ├─ ReplayManifest.cs                         # Manifest schema (schemaVersion 1.0 / 2.0)
 │   ├─ ReplayManifestValidator.cs                # Validation logic
 │   ├─ DeterministicHash.cs                      # Hash + Merkle helpers (via ICryptoHash)
 │   ├─ CanonicalJson.cs                          # Canonical JSON serialization
 │   ├─ Bundle/, Export/, FeedSnapshot/           # Replay bundle writer, export manifest, snapshot coordinator
 │   └─ PolicySimulationInputLock.cs
 ├─ StellaOps.Audit.ReplayToken/                  # Replay token model + generator
 │   ├─ ReplayToken.cs                            # Token model (Value/Algorithm/Version/expiry)
 │   ├─ ReplayTokenRequest.cs                     # Token request inputs
 │   ├─ IReplayTokenGenerator.cs                  # Generator + ReplayTokenVerificationResult enum
 │   └─ Sha256ReplayTokenGenerator.cs            # SHA-256 implementation
 └─ StellaOps.Replay/                             # Shared replay models

2) External dependencies

3) Contracts & data model

3.1 ReplayManifest

The manifest (StellaOps.Replay.Core.ReplayManifest) captures all inputs required to reproduce a verdict. schemaVersion is "1.0" (default, ReplayManifestVersions.V1) or "2.0" (V2); the v2 example below is shown. Top-level members are scan, reachability, and an optional proofSpines array. Reachability-graph references additionally support a legacy sha256 field and namespace / callgraphId properties for v1 compatibility:

{
  "schemaVersion": "2.0",
  "scan": {
    "id": "scan-2025-01-15T10:30:00Z-abc123",
    "time": "2025-01-15T10:30:00.000000Z",
    "policyDigest": "sha256:abc123...",
    "scorePolicyDigest": "sha256:def456...",
    "feedSnapshot": "sha256:789abc...",
    "toolchain": "stellaops/scanner:1.7.3",
    "analyzerSetDigest": "sha256:feed12..."
  },
  "reachability": {
    "analysisId": "reach-xyz",
    "graphs": [
      {
        "kind": "static",
        "casUri": "cas://reachability/graphs/abc123",
        "hash": "blake3:a1b2c3d4...",
        "hashAlg": "blake3-256",
        "analyzer": "elf-callgraph",
        "version": "1.2.0"
      }
    ],
    "runtimeTraces": [],
    "code_id_coverage": {
      "total_nodes": 1500,
      "nodes_with_symbol_id": 1200,
      "nodes_with_code_id": 1100,
      "coverage_percent": 73.33
    }
  },
  "proofSpines": [
    {
      "spineId": "spine-abc123",
      "artifactId": "pkg:npm/lodash@4.17.21",
      "vulnerabilityId": "CVE-2021-23337",
      "verdict": "not_affected",
      "segmentCount": 4,
      "rootHash": "sha256:fedcba...",
      "casUri": "cas://proofs/spines/abc123",
      "createdAt": "2025-01-15T10:30:05.000000Z"
    }
  ]
}

3.2 ReplayToken

A deterministic, content-addressable token (StellaOps.Audit.ReplayToken.ReplayToken). The token value is a SHA-256 hash (hex) computed over the canonicalized ReplayTokenRequest inputs; there is no separate input/output digest pair and no DSSE signature on the token itself:

public sealed partial class ReplayToken : IEquatable<ReplayToken>
{
    public const string Scheme = "replay";
    public const string DefaultAlgorithm = "SHA-256";
    public const string DefaultVersion = "1.0";          // v1.0 = no expiry
    public const string VersionWithExpiration = "2.0";   // v2.0 = includes expiry
    public static readonly TimeSpan DefaultExpiration = TimeSpan.FromHours(1);

    public string Value { get; }                         // SHA-256 hash (hex)
    public string Algorithm { get; }                     // e.g. "SHA-256"
    public string Version { get; }                       // "1.0" or "2.0"
    public DateTimeOffset GeneratedAt { get; }
    public DateTimeOffset? ExpiresAt { get; }            // null => never expires (v1.0)

    // Canonical: "replay:v{Version}:{Algorithm}:{Value}[:{expiryUnixSeconds}]"
    public string Canonical { get; }
}

Token verification re-hashes the supplied inputs and compares against the parsed token, returning a ReplayTokenVerificationResult enum: Valid (0), Invalid (1, hash mismatch), or Expired (2). Verification is performed by IReplayTokenGenerator (Sha256ReplayTokenGenerator).

The inputs that feed the hash are carried by ReplayTokenRequest: FeedManifests, RulesVersion, RulesHash, LatticePolicyVersion, LatticePolicyHash, InputHashes, ScoringConfigVersion, EvidenceHashes, and an AdditionalContext dictionary. The WebService injects AdditionalContext["stellaops.tenant"] from the resolved tenant so a token generated for one tenant cannot be verified under another.

3.3 PolicySimulationInputLock

Captures the pinned SHA-256 digests required for deterministic policy simulation replay (StellaOps.Replay.Core.PolicySimulationInputLock). All digests are SHA-256 hex strings:

public sealed record PolicySimulationInputLock
{
    public required string PolicyBundleSha256 { get; init; }
    public required string GraphSha256 { get; init; }       // dependency graph
    public required string SbomSha256 { get; init; }
    public required string TimeAnchorSha256 { get; init; }   // feed snapshot timestamp proof
    public required string DatasetSha256 { get; init; }      // advisory dataset
    public required DateTimeOffset GeneratedAt { get; init; }
    public bool ShadowIsolation { get; init; }               // require shadow-mode execution
    public string[] RequiredScopes { get; init; } = [];
}

PolicySimulationInputLockValidator.Validate compares the lock against PolicySimulationMaterializedInputs (the inputs materialized at replay time) and a maxAge, returning a PolicySimulationValidationResult whose Reason is one of: ok, policy-bundle-drift, graph-drift, sbom-drift, time-anchor-drift, dataset-drift, shadow-mode-required, missing-scopes, or inputs-lock-stale.

4) REST API (Replay.WebService)

All endpoints are mapped at the service root (no /api prefix). Auth: OpTok (StellaOps resource-server authentication). Tenant context is resolved via the shared StellaOpsTenantResolver; a missing tenant yields 400 missing_tenant.

4.1 Replay tokens (root, Program.cs)

POST /v1/replay/tokens                 GenerateReplayToken  → 201 GenerateTokenResponse  [write]
POST /v1/replay/tokens/verify          VerifyReplayToken    → 200 VerifyTokenResponse     [read]
GET  /v1/replay/tokens/{tokenCanonical} GetReplayToken      → 200 TokenInfoResponse / 404 [read]

Token issuance and verification resolve tenant context through the shared StellaOps tenant resolver and stamp AdditionalContext["stellaops.tenant"] into the hashed request, so a token generated for one tenant cannot be verified under another tenant by swapping request headers.

4.2 Verdict replay (/v1/replay/verdict, VerdictReplayEndpoints.cs)

POST /v1/replay/verdict                ExecuteVerdictReplay  → 200 VerdictReplayResponse     [read]
POST /v1/replay/verdict/verify         VerifyReplayEligibility → 200 ReplayEligibilityResponse [read]
GET  /v1/replay/verdict/{manifestId}/status GetReplayStatus  → 501 (fail-closed)              [read]
POST /v1/replay/verdict/compare        CompareReplayResults  → 200 ReplayComparisonResponse   [read]
POST /v1/replay/verdict/bundle         ProduceVerdictReplayBundle → 200 / 400 / 501 / 502       [write]
POST /v1/replay/verdict/by-subject     ReplayVerdictBySubject → 200 / 400 / 404 / 501 / 502     [read]

POST /v1/replay/verdict reads an audit bundle (IAuditBundleReader), checks eligibility (IVerdictReplayPredicate), then executes an isolated, offline replay (IReplayExecutor) and reports verdict/decision match plus any drifts.

GET /v1/replay/verdict/{manifestId}/status is intentionally fail-closed until durable replay execution history exists. The endpoint returns 501 problem+json (replay_status_unavailable) instead of a synthetic not_replayed status placeholder.

POST /v1/replay/verdict/bundle is the path-oriented operator workflow: it captures the current Policy.Engine decision, seals an AuditPack bundle, and returns the server-local bundlePath needed by a subsequent explicit replay. POST /v1/replay/verdict/by-subject is the Console adapter. It composes bundle production and replay in-process, but its public bundle receipt contains only bundleId, decision/digest fields, and the replay result; the server-local path is deliberately omitted. This deterministic recomputation is a read/verification operation; the temporary internal bundle does not grant durable replay production. Before producing a bundle, the adapter verifies the exact packId / version / canonical subject tuple against Release Orchestrator’s tenant-scoped recorded gate-decision projection. An unrecorded or non-canonical subject returns 404 replay_subject_not_found; verification failure returns a stable problem and writes no bundle. This prevents an arbitrary new subject from being evaluated, sealed, and compared with itself as a tautological Match. Policy.Engine transport failures return stable 502 problem details and never copy downstream bodies, exception messages, or local filesystem paths into the client response.

4.3 Point-in-time advisory + feed snapshots (PointInTimeQueryEndpoints.cs)

GET  /v1/pit/advisory/{cveId}              QueryAdvisoryAtPointInTime → 200 / 404 / 400  [read]
POST /v1/pit/advisory/cross-provider       QueryCrossProviderAdvisory → 200 / 400        [read]
GET  /v1/pit/advisory/{cveId}/timeline     GetAdvisoryTimeline        → 200 / 404        [read]
POST /v1/pit/advisory/diff                 CompareAdvisoryAtTimes     → 200 / 400        [read]

POST /v1/pit/snapshots                     CaptureFeedSnapshot        → 201 / 400        [write] (audited)
GET  /v1/pit/snapshots/{digest}            GetFeedSnapshot            → 200 / 404        [write]
GET  /v1/pit/snapshots/{digest}/verify     VerifySnapshotIntegrity    → 200 / 404        [write]
POST /v1/pit/snapshots/bundle              CreateSnapshotBundle       → 200 / 400        [write]

CaptureFeedSnapshot is wrapped in an audit emission (AuditModules.Replay / AuditActions.Replay.CaptureFeedSnapshot).

4.4 Operational endpoints

GET  /healthz                              health check
GET  /.well-known/openapi                  ReplayOpenApiDocument (OpenAPI 3.1.0, tokens only)
GET  /.well-known/buildinfo                build/provenance info (MapBuildInfoEndpoint)

There is no /readyz endpoint, and there are no GET /manifests/{scanId} or POST /manifests endpoints.

Authorization policies (ReplayAuthorizationPolicies.cs). Two ASP.NET policies are registered: replay.token.read requires the canonical replay:read scope and replay.token.write requires replay:write. These are policy names, not scope strings. Read includes deterministic execution, eligibility, comparison, status, token verification, and the Console by-subject verification adapter. Write remains required for explicit durable token, bundle, manifest, and feed-snapshot production. The resource server clears its global RequiredScopes collection because a global scope would be ANDed with every endpoint policy and collapse this least-privilege split.

5) Configuration (YAML)

Configuration is loaded via AddStellaOpsDefaults with environment prefix REPLAY_, plus an optional ../etc/replay.yaml. The Replay:Authority section binds to AuthorityConfig; storage is resolved by the Storage / Replay:Storage keys read in Program.cs. Defaults shown below come from AuthorityConfig and the storage resolvers.

Replay:
  Authority:
    Issuer: "https://authority.stella-ops.local"               # AuthorityConfig.Issuer default
    RequireHttpsMetadata: true
    MetadataAddress: "https://authority.stella-ops.local/.well-known/openid-configuration"
    Audiences: ["stella-ops-api"]                               # default audience
    # No global RequiredScopes: endpoint read/write policies are authoritative.

  PolicyEngine:
    Enabled: false                                               # fail-closed 501 when disabled
    BaseAddress: "http://policy-engine:8080"                    # required when enabled
    TimeoutSeconds: 30
    ServiceIdentity:
      Enabled: false                                             # opt-in service-token flow
      ClientId: "stellaops-release-dispatch"
      ClientSecret: null                                         # deployment secret, never committed

  SubjectVerification:
    Enabled: false                                               # fail-closed until RO verification is configured
    BaseAddress: "http://release-orchestrator.stella-ops.local:8080"
    TimeoutSeconds: 10
    ServiceTokenScope: "release:read"

  # Storage split (Sprint 312): postgres index + seed-fs blobs.
  Storage:
    Driver: "postgres"                                          # default; "inmemory" Testing-only
    Postgres:
      ConnectionString: "Host=postgres;Database=replay;..."     # or ConnectionStrings:Default
    ObjectStore:
      Driver: "seed-fs"                                         # only seed-fs supported; rustfs rejected
      SeedFs:
        RootPath: "data/replay/snapshots"                       # default blob root

Notes grounded in Program.cs:

6) Determinism guarantees

6.1 Input pinning

All inputs that affect verdict output are captured. In the ReplayManifest schema, policy/score digests and toolchain/analyzer-set identifiers are stored inline on scan, while reachability graphs and proof spines carry a casUri plus hash. The runtime feed-snapshot store is not a generic CAS: blob payloads live in the seed-fs object store and are indexed by digest in replay.feed_snapshot_index (PostgreSQL).

InputPinning MethodStorage
Policy YAMLContent digest (scan.policyDigest)Inline in manifest
Score policyContent digest (scan.scorePolicyDigest)Inline in manifest
Feed snapshotSnapshot digest + timestampseed-fs blob + Postgres index
ToolchainImage reference (scan.toolchain)Inline in manifest
Analyzersscan.analyzerSetDigestInline in manifest
Reachability graphshash + hashAlg (BLAKE3-256 default) + casUriCAS reference in manifest

6.2 Output determinism

GuaranteeImplementation
Canonical JSONSorted keys, no whitespace variation
Stable orderingDeterministic sort on all collections
UTC timestampsMicrosecond precision, always UTC
Hash stabilitySame input → same hash

6.2a Determinism contract (clock injection)

Replay’s verifier writes a VerifiedAt field on every VerificationResult. Because Replay’s core promise is “two runs over the same input produce byte-identical output”, that timestamp must come from an injected TimeProvider, never from DateTimeOffset.UtcNow. As of SPRINT_20260501_042 (REPLAY-DET-002), DeterminismVerifier accepts a TimeProvider constructor parameter:

The Roslyn analyzer STELLA0112 (DateTimeOffset.UtcNow) enforces this for all production code paths under src/Replay/**. Operational correlation identifiers — ReplayExecutor.GenerateReplayId and ReplayJobQueue.GenerateJobId — are explicit [StellaOps.Determinism.DeterminismAllowed("…")] exceptions (each carrying a written reason) because they label individual in-flight executions, not content-addressed audit state; the determinism contract is enforced over the verdict digest (DeterminismVerifier.ComputeVerdictDigest), not over those labels.

See: docs/implplan/SPRINT_20260501_040_Determinism_roslyn_analyzer.md, docs/implplan/SPRINT_20260501_041_Determinism_provider_abstractions.md, docs/implplan/SPRINT_20260501_042_Determinism_sweep_replay.md.

6.3 TLPT baseline freeze

Replay Core exposes a TLPT baseline contract for DORA TLPT preparation: tlpt-baseline.v1. The core producer builds a canonical manifest that binds a tlpt-scope.v1 scopeHash to frozen feed, policy, dependency graph, and asset-registry state references.

The baseline manifest includes:

StateFrozen fields
FeedsProvider snapshot digests, captured timestamps, epoch timestamps, and a computed feed bundle hash
PolicyPolicy id, version, digest, and captured timestamp
Dependency graphGraph id, source, digest, and captured timestamp
Asset stateAsset source, schema version, digest, and captured timestamp

baselineHash is computed over canonical JSON excluding baselineHash. Offline verification recomputes the feed bundle hash and manifest hash without network access. ReplayFrozenState rehydrates only the state embedded in the verified baseline, so later Replay consumers receive identical state references for the same baseline.

7) Security & compliance

8) Performance targets

9) Observability

The WebService wires standard StellaOps telemetry via AddStellaOpsTelemetry(..., "replay-webservice"), which provides OpenTelemetry ASP.NET Core, HTTP-client, and runtime auto-instrumentation plus the OTLP exporter. Request logging is emitted through Serilog (UseSerilogRequestLogging).

Not implemented (proposed). The custom domain metrics and spans below are a target design — there is currently no dedicated Meter or ActivitySource in src/Replay/** emitting these names. Treat them as a roadmap until the instrumentation lands.

  • replay.tokens.issued_total{issuer}
  • replay.tokens.verified_total{result=valid|invalid}
  • replay.manifests.stored_total
  • replay.verification.duration_seconds
  • Spans for token operations, manifest storage, and verification workflows.

10) Testing matrix

11) Storage contract (Sprint 312)

Replay now follows the platform storage split used by Scanner:

The embedded migration (Migrations/001_initial_schema.sql) creates the replay schema and a single table (the only persisted table the WebService owns today):

CREATE SCHEMA IF NOT EXISTS replay;

CREATE TABLE IF NOT EXISTS replay.feed_snapshot_index (
    provider_id     TEXT NOT NULL,
    digest          TEXT NOT NULL,
    captured_at     TIMESTAMPTZ NOT NULL,
    epoch_timestamp TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (provider_id, captured_at, digest)
);

CREATE INDEX IF NOT EXISTS idx_replay_snapshot_index_lookup
    ON replay.feed_snapshot_index (provider_id, captured_at DESC, digest ASC);

Inserts use ON CONFLICT (provider_id, captured_at, digest) DO NOTHING (append-only). Snapshot blob payloads are stored on disk by SeedFsFeedSnapshotBlobStore as {key}.bin + {key}.meta.json under the configured root path, keyed by a filesystem-safe form of the digest.

Verification evidence: