Attestor Observability Runbook

Audience: Attestor operators and SREs running the service in connected or air-gapped deployments.
Purpose: Describe the dashboards, metrics, logs, health endpoints, and alert thresholds used to monitor Stella Ops Attestor, including offline verification steps.

Dashboards (offline import)

Grafana JSON: dashboards/attestor-observability.json. Import locally; no external data sources are assumed.
Panels: signing latency p95/p99, verification failure rate, transparency log submission lag, key rotation age, DSSE verification errors, queue depth/backlog, and attestation bundle size histogram.

Key metrics

Metric	Description
`attestor.sign_latency_seconds`	Signing latency per request.
`attestor.verify_total{result}`	Verification attempts, including failures by result.
`attestor.submit_latency_seconds`	Transparency-log submission latency.
`attestor.rekor_queue_depth`	Pending/retrying Rekor submission jobs.
`attestor.rekor_dead_letter_total`	Rekor submissions moved to dead letter.

Logs and traces

Correlate by correlationId, attestationId, tenant, and keyId. Include policyVersion, tlogIndex, and rekorUrl where applicable.
Traces are disabled by default for air-gapped deployments. Enable them by pointing the OTLP exporter at an on-prem collector.

Health and diagnostics

/health/live and /health/ready expose the current ASP.NET health-check set.
/buildinfo.json (also /api/v1/buildinfo) exposes baked image provenance; compare its digest and commit with the offline bundle manifest.
Verification probe: authenticated POST /api/v1/rekor/verify with a known frozen receipt/proof fixture.

Alert hints

Condition	Suggested threshold
Signing latency (p99)	> 1s
Verification failures	Sustained spike above baseline
Transparency log submission lag	> 10s
Key rotation age	Exceeds policy threshold
Queue backlog	Above configured threshold

Offline verification steps

Import the Grafana JSON locally and point it at a Prometheus instance scraping the attestor target.
Run a sample verification and confirm metrics and logs emit locally.
Fetch /buildinfo.json and compare the commit, version, and image provenance against the offline bundle manifest.

Bundle Rotation Operations Guide
Attestor Architecture
Offline Verification Guide
Dashboard: dashboards/attestor-observability.json