Fidelity Metrics Framework

Sprint: SPRINT_3403_0001_0001_fidelity_metrics

This document describes the three-tier fidelity metrics framework for measuring deterministic reproducibility in StellaOps scanner outputs.

Overview

Fidelity metrics quantify how consistently the scanner produces outputs across replay runs. The framework provides three tiers of measurement, each capturing different aspects of reproducibility:

MetricAbbrev.DescriptionTarget
Bitwise FidelityBFByte-for-byte identical outputs≥ 0.98
Semantic FidelitySFNormalized object equivalence≥ 0.99
Policy FidelityPFPolicy decision consistency≈ 1.0

Metric Definitions

Bitwise Fidelity (BF)

Measures the proportion of replay runs that produce byte-for-byte identical outputs.

BF = identical_outputs / total_replays

What it captures:

When BF < 1.0:

Semantic Fidelity (SF)

Measures the proportion of replay runs that produce semantically equivalent outputs, ignoring formatting differences.

SF = semantic_matches / total_replays

What it compares:

When SF < 1.0 but BF = SF:

When SF < 1.0:

Policy Fidelity (PF)

Measures the proportion of replay runs that produce matching policy decisions.

PF = policy_matches / total_replays

What it compares:

When PF < 1.0:

Prometheus Metrics

The fidelity framework exports the following metrics:

Metric NameTypeLabelsDescription
fidelity_bitwise_ratioGaugetenant_id, surface_idBitwise fidelity ratio
fidelity_semantic_ratioGaugetenant_id, surface_idSemantic fidelity ratio
fidelity_policy_ratioGaugetenant_id, surface_idPolicy fidelity ratio
fidelity_total_replaysGaugetenant_id, surface_idNumber of replays
fidelity_slo_breach_totalCounterbreach_type, tenant_idSLO breach count

SLO Thresholds

Default SLO thresholds (configurable):

MetricWarningCritical
Bitwise Fidelity< 0.98< 0.90
Semantic Fidelity< 0.99< 0.95
Policy Fidelity< 1.0< 0.99

Integration with DeterminismReport

Fidelity metrics are integrated into the DeterminismReport record:

public sealed record DeterminismReport(
    // ... existing fields ...
    FidelityMetrics? Fidelity = null);

public sealed record DeterminismImageReport(
    // ... existing fields ...
    FidelityMetrics? Fidelity = null);

Usage Example

// Create fidelity metrics service
var service = new FidelityMetricsService(
    new BitwiseFidelityCalculator(),
    new SemanticFidelityCalculator(),
    new PolicyFidelityCalculator());

// Compute fidelity from baseline and replays
var baseline = LoadScanResult("scan-baseline.json");
var replays = LoadReplayScanResults();
var fidelity = service.Compute(baseline, replays);

// Check thresholds
if (fidelity.BitwiseFidelity < 0.98)
{
    logger.LogWarning("BF below threshold: {BF}", fidelity.BitwiseFidelity);
}

// Include in determinism report
var report = new DeterminismReport(
    // ... other fields ...
    Fidelity: fidelity);

Mismatch Diagnostics

When fidelity is below threshold, the framework provides diagnostic information:

public sealed record FidelityMismatch
{
    public required int RunIndex { get; init; }
    public required FidelityMismatchType Type { get; init; }
    public required string Description { get; init; }
    public IReadOnlyList<string>? AffectedArtifacts { get; init; }
}

public enum FidelityMismatchType
{
    BitwiseOnly,    // Hash differs but content equivalent
    SemanticOnly,   // Content differs but policy matches
    PolicyDrift     // Policy decision differs
}

Configuration

Configure fidelity options via FidelityThresholds:

{
  "Fidelity": {
    "BitwiseThreshold": 0.98,
    "SemanticThreshold": 0.99,
    "PolicyThreshold": 1.0,
    "EnableDiagnostics": true,
    "MaxMismatchesRecorded": 100
  }
}

Source Files