Notify Observability Runbook

Audience: operators and on-call engineers monitoring the Notify service — delivery health, queue pressure, and tenant isolation.

This runbook lists the dashboards, metrics, logs, traces, and health probes used to operate Notify, including offline/air-gap import steps for the bundled Grafana assets.

Dashboards (offline import)

Grafana JSON: dashboards/notify-observability.json — import locally; no external data sources are assumed.
Panels: enqueue/dequeue rate, delivery latency p95/p99, channel error rate, retry/dead-letter counts, rule-evaluation latency, tenant-isolation breaches (should stay 0), and notification-simulation probe failures.

Key metrics

notify.delivery.attempts{tenant_id,channel_type,status} - delivery attempts recorded by the worker event processor when a channel delivery is queued.
notify.delivery.duration{tenant_id,channel_type,status} - delivery append latency in milliseconds.
notify.rule.evaluations{tenant_id,rule_id,matched} - rule evaluations recorded for each enabled rule.
notify.rule.evaluation.duration{tenant_id,rule_id,matched} - rule evaluation latency in milliseconds.
notify.deadletter.entries{tenant_id,reason,channel_type} - dead-letter entries.
notify.queue.depth{tenant_id,channel_type} - current queue depth per channel.

Logs and traces

Correlate by notificationId, ruleId, tenant, and channel. Include the quietHoursApplied, correlationKey, and retries fields.
Traces are disabled by default for air-gap. Enable them by pointing the OTLP exporter at an on-prem collector.

Health and diagnostics

/health/liveness and /health/readiness check queue-backend reachability and channel-provider credentials.
/status exposes the build version, commit, and feature flags; verify these against the offline bundle manifest.
Simulation probe: POST /api/v2/simulate with a sample rule set validates correlation/digest wiring without side effects. See ../api-reference.mdfor the request/response contract.

Alert hints

Delivery latency p99 > 1.5s for email/webhook channels.
Dead-letter queue growth above threshold.
Rule-evaluation latency p99 > 500ms.
Correlation/quiet-hours simulation failures.

Prometheus alert rules ship alongside this runbook in alerts/notify-slo-alerts.yaml.

Offline verification steps

Import the Grafana JSON locally and point it at the Prometheus scrape labelled notify.
Run a simulation via POST /api/v2/simulate with a sample rule set and confirm the response body and structured logs emit locally.
Fetch /status and compare the commit/version to the offline bundle manifest.