Scanner observability

Verified against commit 942a18727c (2026-09-26). Re-verify with: grep -rhoE "Create(Counter|Histogram|UpDownCounter|ObservableGauge)<[a-z]+>\(\s*\"[^\"]+\"" src/Scanner src/__Libraries/StellaOps.Scanner.ProofSpine and grep -oE "\"scanner_worker_[a-z_]+\"" src/Scanner/StellaOps.Scanner.Worker/Diagnostics/ScannerWorkerMetrics.cs.

This page records the metric, log, and trace contracts that operators bind dashboards and alerts to. Renaming or removing a metric listed here requires a sprint task and a same-change update of the dashboards under operations/.

Metric inventory

Metric	Type	Source
`scanner_queue_enqueued_total`, `scanner_queue_deduplicated_total`, `scanner_queue_ack_total`, `scanner_queue_retry_total`, `scanner_queue_deadletter_total`	counters	`src/Scanner/__Libraries/StellaOps.Scanner.Queue/QueueMetrics.cs`
`scanner.layer_cache_hits_total`, `scanner.layer_cache_misses_total`, `scanner.layer_cache_evictions_total`, `scanner.layer_cache_bytes`	counters / histogram	`src/Scanner/__Libraries/StellaOps.Scanner.Cache/ScannerCacheMetrics.cs`
`scanner.file_cas_hits_total`, `scanner.file_cas_misses_total`, `scanner.file_cas_evictions_total`, `scanner.file_cas_bytes`	counters / histogram	same
`scanner_worker_job_duration_ms`, `scanner_worker_stage_duration_ms`, `scanner_worker_queue_latency_ms`	histograms	`src/Scanner/StellaOps.Scanner.Worker/Diagnostics/ScannerWorkerMetrics.cs`
`scanner_worker_jobs_completed_total`, `scanner_worker_jobs_failed_total`	counters	same
`scanner_worker_language_cache_{hits,misses}_total`, `scanner_worker_os_cache_{hits,misses}_total`	counters	same
`scanner_worker_surface_manifests_{published,failed,skipped}_total`, `scanner_worker_surface_manifest_publish_duration_ms`, `scanner_worker_surface_payload_persisted_total`	counters / histogram	same
`scanner_worker_registry_secret_requests_total`, `scanner_worker_registry_secret_ttl_seconds`	counter / histogram	same
`scanner_worker_secrets_analysis_{completed,failed}_total`, `scanner_worker_secrets_analysis_duration_ms`, `scanner_worker_secrets_findings_detected_total`	counters / histogram	same
`scanner_analyzer_golang_heuristic_total{indicator,version_hint}`	counter	`src/Scanner/__Libraries/StellaOps.Scanner.Analyzers.Lang.Go/Internal/GoAnalyzerMetrics.cs`
`entrytrace_resolutions_total{outcome}`, `entrytrace_unresolved_total{reason}`	counters	`src/Scanner/__Libraries/StellaOps.Scanner.EntryTrace/Diagnostics/EntryTraceMetrics.cs`
`scanner.runtime.policy.requests`, `scanner.runtime.policy.latency.ms`	counter / histogram	`src/Scanner/StellaOps.Scanner.WebService/Services/RuntimePolicyService.cs`
`scanner_proofspine_validation_failures_total`, `scanner_proofspine_validation_successes_total`	counters	`src/__Libraries/StellaOps.Scanner.ProofSpine/Diagnostics/ProofSpineValidationMetrics.cs` (details below)

Scanner has no /metrics HTTP route (Q-25). The Worker exports metrics and traces through OpenTelemetry (src/Scanner/StellaOps.Scanner.Worker/Diagnostics/TelemetryExtensions.cs: OTLP when an endpoint is configured, console otherwise); the WebService telemetry switches live under scanner:telemetry.

ProofSpine validation metrics

`scanner_proofspine_validation_failures_total`

Counter. Incremented once per ProofSpine input rejected by StellaOps.Scanner.ProofSpine.Validation.ProofSegmentValidator (src/__Libraries/StellaOps.Scanner.ProofSpine/Validation/ProofSegmentValidator.cs).

Sprint reference: SPRINT_20260429_003_Scanner_proofspine_input_validation.md.

Label	Cardinality	Values
`reasonCode`	Bounded (≤16)	One of the codes listed in proofspine.md.
`tenantId`	Bounded by tenant population	Tenant the request belongs to, or `unknown`.

Combined cardinality stays well below the 100-label threshold defined in .memory/standards/TESTING_PRACTICES.md (“Metrics contracts”).

`scanner_proofspine_validation_successes_total`

Counter. Paired success counter, used by dashboards to compute failure rate failures / (failures + successes). Single label tenantId.

Sample Grafana panel

Panel JSON (paste into a Grafana dashboard’s “Add panel -> Import JSON” dialog):

{
  "title": "ProofSpine validation failure rate",
  "type": "timeseries",
  "datasource": "Prometheus",
  "targets": [
    {
      "refId": "A",
      "expr": "sum by (reasonCode) (rate(scanner_proofspine_validation_failures_total[5m]))",
      "legendFormat": "{{reasonCode}}"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "ops",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.1 },
          { "color": "red", "value": 1 }
        ]
      }
    }
  },
  "options": {
    "legend": { "displayMode": "table", "placement": "right" },
    "tooltip": { "mode": "multi" }
  }
}

A companion ratio panel (failures over total):

{
  "title": "ProofSpine validation failure ratio",
  "type": "stat",
  "datasource": "Prometheus",
  "targets": [
    {
      "refId": "A",
      "expr": "sum(rate(scanner_proofspine_validation_failures_total[5m])) / clamp_min(sum(rate(scanner_proofspine_validation_failures_total[5m]) + rate(scanner_proofspine_validation_successes_total[5m])), 1)"
    }
  ],
  "fieldConfig": {
    "defaults": { "unit": "percentunit" }
  }
}

Recommended alerts

sum(rate(scanner_proofspine_validation_failures_total{reasonCode!~"segment_status_invalid|segments_property_missing"}[5m])) > 0.5 for 10 minutes - production proof spines should not be rejected at this rate.
sum(rate(scanner_proofspine_validation_failures_total{reasonCode="segments_too_many"}[5m])) > 0 for 1 minute - a single hit of segments_too_many indicates an attempt to abuse the algorithmic-complexity ceiling.

Logs

The validator itself does not emit logs (it is a pure function). The endpoint records validation outcomes via the metrics counter. Operators correlate spikes in the failure counter with the request log for the matching tenantId and the X-Correlation-Id header.

Traces

ProofSpine endpoints inherit the standard Scanner WebService OTel span set: HTTP GET /api/v1/spines/{spineId} with attributes tenant.id, correlation.id, and (on failure) proofspine.validation.reasonCode. The validator does not start its own span; it adds the reason code as an attribute on the parent endpoint span when called.