Scanner observability

This page records the metric, log, and trace contracts that operators bind dashboards and alerts to. Counters and labels listed here are stable; renaming or removing them requires a sprint task and a deprecation period.

Metrics

scanner_proofspine_validation_failures_total

Counter. Incremented once per ProofSpine input rejected by StellaOps.Scanner.ProofSpine.Validation.ProofSegmentValidator.

Sprint reference: SPRINT_20260429_003_Scanner_proofspine_input_validation.md.

Combined cardinality stays well below the 100-label threshold defined in docs/code-of-conduct/TESTING_PRACTICES.md (“Metrics contracts”).

scanner_proofspine_validation_successes_total

Counter. Paired success counter, used by dashboards to compute failure rate failures / (failures + successes). Single label tenantId.

Sample Grafana panel

Panel JSON (paste into a Grafana dashboard’s “Add panel -> Import JSON” dialog):

{
  "title": "ProofSpine validation failure rate",
  "type": "timeseries",
  "datasource": "Prometheus",
  "targets": [
    {
      "refId": "A",
      "expr": "sum by (reasonCode) (rate(scanner_proofspine_validation_failures_total[5m]))",
      "legendFormat": "{{reasonCode}}"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "ops",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.1 },
          { "color": "red", "value": 1 }
        ]
      }
    }
  },
  "options": {
    "legend": { "displayMode": "table", "placement": "right" },
    "tooltip": { "mode": "multi" }
  }
}

A companion ratio panel (failures over total):

{
  "title": "ProofSpine validation failure ratio",
  "type": "stat",
  "datasource": "Prometheus",
  "targets": [
    {
      "refId": "A",
      "expr": "sum(rate(scanner_proofspine_validation_failures_total[5m])) / clamp_min(sum(rate(scanner_proofspine_validation_failures_total[5m]) + rate(scanner_proofspine_validation_successes_total[5m])), 1)"
    }
  ],
  "fieldConfig": {
    "defaults": { "unit": "percentunit" }
  }
}

Recommended alerts

• sum(rate(scanner_proofspine_validation_failures_total{reasonCode!~"segment_status_invalid|segments_property_missing"}[5m])) > 0.5 for 10 minutes - production proof spines should not be rejected at this rate.
• sum(rate(scanner_proofspine_validation_failures_total{reasonCode="segments_too_many"}[5m])) > 0 for 1 minute - a single hit of segments_too_many indicates an attempt to abuse the algorithmic-complexity ceiling.

Logs

The validator itself does not emit logs (it is a pure function). The endpoint records validation outcomes via the metrics counter. Operators correlate spikes in the failure counter with the request log for the matching tenantId and the X-Correlation-Id header.

Traces

ProofSpine endpoints inherit the standard Scanner WebService OTel span set: HTTP GET /api/v1/spines/{spineId} with attributes tenant.id, correlation.id, and (on failure) proofspine.validation.reasonCode. The validator does not start its own span; it adds the reason code as an attribute on the parent endpoint span when called.