Attestor Observability Runbook

Audience: operators and SREs running the attestation surface (hosted by evidence-web / evidence-worker since 2026-09-05) in connected or air-gapped deployments.
Purpose: Describe the dashboards, metrics, logs, health endpoints, and alert thresholds used to monitor Stella Ops Attestor, including offline verification steps.

Dashboards (offline import)

Grafana JSON: dashboards/attestor-observability.json. Import locally; no external data sources are assumed.
Panels: signing latency p95/p99, verification failure rate, transparency log submission lag, key rotation age, DSSE verification errors, queue depth/backlog, and attestation bundle size histogram.

Key metrics

Metric	Description
`attestor.sign_latency_seconds`	Signing latency per request.
`attestor.verify_total{result}`	Verification attempts, including failures by result.
`attestor.submit_latency_seconds`	Transparency-log submission latency.
`attestor.rekor_queue_depth`	Pending/retrying Rekor submission jobs.
`attestor.rekor_dead_letter_total`	Rekor submissions moved to dead letter.

Logs and traces

Correlate by correlationId, attestationId, tenant, and keyId. Include policyVersion, tlogIndex, and rekorUrl where applicable.
Traces are disabled by default for air-gapped deployments. Enable them by pointing the OTLP exporter at an on-prem collector.

Health and diagnostics

The attestation surface runs inside evidence-web; its readiness route is GET /health/ready (src/Evidence/StellaOps.Evidence.WebService/Program.cs). Host-level health, Doctor and build-info routes are documented in ../../evidence/architecture.md.
Verification probe: authenticated POST /api/evidence/v1/rekor/verify with a known frozen receipt/proof fixture.

Alert hints

Condition	Suggested threshold
Signing latency (p99)	> 1s
Verification failures	Sustained spike above baseline
Transparency log submission lag	> 10s
Key rotation age	Exceeds policy threshold
Queue backlog	Above configured threshold

Offline verification steps

Import the Grafana JSON locally and point it at a Prometheus instance scraping the attestor target.
Run a sample verification and confirm metrics and logs emit locally.
Fetch /buildinfo.json and compare the commit, version, and image provenance against the offline bundle manifest.

Bundle Rotation Operations Guide
Attestor Architecture
Offline Verification Guide
Dashboard: dashboards/attestor-observability.json