Notifier Runbook

Last updated: 2026-06-04 (Notify connector bundle readiness update)

Purpose

Operational steps to deploy, monitor, and recover the Notifier service. Notifier is the deployable host (WebService + Worker) that composes the shared Notify toolkit (src/Notify) into the Notifications Studio. It owns the PostgreSQL notify schema and the Valkey/Redis (or NATS JetStream) notification queue.

Audience: operators and on-call engineers who deploy, monitor, and troubleshoot the Notifier service in a Stella Ops deployment.

• WebService (StellaOps.Notifier.WebService): management/admin REST API, rule and template CRUD, simulation, delivery browsing, dead-letter administration, storm-breaker status, webhook secret rotation.
• Worker (StellaOps.Notifier.Worker): queue-driven dispatch, channel delivery, rule evaluation, correlation, storm-breaking (dedup/throttle), escalation, digest, dead-letter handling, and retention cleanup.

Companion Notify web-only service (src/Notify, slot 29) shares the same notify schema and toolkit; this runbook targets the Notifier host (slot 28).

DEPLOYMENT NOTE (verify against the compose you run): in the current active stack (devops/compose/docker-compose.stella-services.yml), Slot 28’s WebService container (notifier-web) is commented out / merged into notify-web— only notifier-worker runs as a standalone Notifier deployable. So although the StellaOps.Notifier.WebService codebase below is real and owns the management API surface, in that profile the admin/ management REST routes are served by the consolidated notify-web (slot 29) and the Notifier ships only its worker. The notifier-web block is retained (commented) for rollback. The legacy split-container profile (docker-compose.stella-ops.legacy.yml) behaves the same way.

Pre-flight

• Authority client registered for Notifier with the appropriate scopes: notify.viewer (read), notify.operator (rule/channel/delivery operations), notify.admin (secrets, escalations, platform settings), notify.escalate. (Defined in StellaOps.Auth.Abstractions/StellaOpsScopes.cs.)
• Channel credentials (SMTP, Slack/Teams/PagerDuty/OpsGenie, webhook HMAC keys) are stored as per-channel rows in the notify schema, not as service env vars. The dev-only SMTP override (NOTIFIER_NOTIFIER__EMAIL__SMTP__* → config section Notifier:Email:Smtp) is the exception for local capture (Mailpit).
• Outbound egress controlled by the AirGap egress policy (AddAirGapEgressPolicy); ensure target channel hosts are permitted under the active egress allowlist. (Inbound webhook IP allowlists are a separate per-channel control.)
• PostgreSQL and Valkey/Redis reachable; the notify schema auto-migrates on startup (forward-only; see ADR-004). Queue health check (AddNotifyQueueHealthCheck) passes.
• Offline kit loaded: channel manifests, default templates (devops/offline/notifier/templates/, with a build-time copy embedded under src/Notifier/.../WebService/offline/notifier/templates/; there is no repo-root offline/ directory), rule seeds. Clean default startup does NOT seed demo channels/templates/rules/incidents: the shared Notify persistence registration defaults runSeedMigrations to false, and the Notifier host (StellaOps.Notifier.WebService/.Worker) calls AddNotifyPersistence(...) without overriding it. The NOTIFY_BOOTSTRAP_ENABLED=true toggle that opts demo seeding back in is read by the companion Notify web service (slot 29) — not by the Notifier host — so it has no effect here.
• Companion Notify.WebService plug-in readiness requires the base Notify bundle mounted under notify:plugins:directory. The required base bundle contains only StellaOps.Notify.Connectors.InApp and StellaOps.Notify.Connectors.InAppInbox; optional email/chat/paging/regulatory connector overlays do not belong in the base stack.
• The companion Notify.WebService container reads the runtime config from /app/etc/notify/notify.yaml; compose mounts devops/etc/notify/ read-only at /app/etc/notify, so place the selected notify.*.yaml template there as notify.yaml before startup.

Deploy

1. Apply the supported Compose or Offline Kit profile with image digests pinned. The Notifier images are stellaops/notifier-web and stellaops/notifier-worker. In the current active compose only stellaops-notifier-worker is enabled; the notifier-web container is commented out (its API surface is served by the consolidated notify-web, slot 29). Deploy notifier-web separately only if you uncomment that block.
2. Set config (env-var prefix NOTIFIER_; __ maps to config section nesting):
   • NOTIFIER_NOTIFIER__STORAGE__POSTGRES__CONNECTIONSTRING (section notifier:storage:postgres)
   • NOTIFIER_NOTIFIER__QUEUE__TRANSPORT = redis (or nats) and NOTIFIER_NOTIFIER__QUEUE__REDIS__CONNECTIONSTRING (section notifier:queue)
   • Authority/Router auth wiring via the standard Router / Authority sections.
   • ASPNETCORE_URLS for the WebService listener (default container port 8080; local dev https://localhost:10280 / http://localhost:10281).
3. Verify liveness/readiness: WebService GET /healthz returns 200 Healthy (ASP.NET health-check format, not a JSON ready flag). For the companion notify-web host, GET /readyz returns 503 with missing base connector names when the InApp/InAppInbox base plug-ins are absent or mounted at the wrong notify:plugins:directory. The Worker exposes its own worker health endpoints via MapWorkerHealthEndpoints (includes the queue health check).

NOT IMPLEMENTED: there is no warm-cache admin endpoint (POST .../admin/warm); rules and templates are loaded lazily/per-request and from durable storage.

Monitor

The Worker emits OpenTelemetry instruments via the StellaOps.Notifier meter. Instrument names (dot-notation as emitted; a Prometheus exporter renders these with _/_total suffix conventions):

• Delivery: notify.delivery.attempts (counter, tags: tenant/channel/status), notify.delivery.duration (histogram, ms). Notifier-summary family also emits notifier.delivery.attempts / .successes / .failures and notifier.delivery.latency.
• Escalation: notify.escalation.events; notifier.escalation.ack_latency.
• Rule evaluation: notify.rule.evaluations, notify.rule.evaluation.duration.
• Templates: notify.template.renders, notify.template.render.duration.
• Storm/dedup: notify.storm.events; notifier.storm.events, notifier.suppression.count.
• Dead-letter: notify.deadletter.entries; notifier.deadletter.count.
• Queue depth: notify.queue.depth (observable gauge, per channel); retention via notify.retention.cleanups.

SLO alerting rules ship in docs/notifications/operations/alerts/notify-slo-alerts.yaml and reference the exporter metric names notify_delivery_success_total, notify_delivery_total, notify_backlog_depth, and notify_delivery_duration_seconds_bucket. CAVEAT: those alert-rule names do NOT map 1:1 onto the instruments the worker actually emits — the code emits notify.delivery.attempts (tagged by status, not a separate success counter), notify.delivery.duration, and notify.queue.depth (per-channel gauge), with no instrument literally named notify.delivery.success, notify.delivery (total), or notify.backlog.depth. Wiring the alerts to live data therefore requires recording rules / relabeling that derive the alert-rule series from the emitted instruments; the shipped rules are not satisfiable directly against the raw exporter output.

• Logs: structured console JSON with UTC timestamps; preserve/propagate X-StellaOps-Correlation-Id (GUID/ULID) across inbound and outbound calls.
• Traces: span notify.delivery (and notify.escalation) from ActivitySource StellaOps.Notifier.

Common operations

All management routes are under /api/v2/notify (v1 routes are deprecated and emit Deprecation/Sunset headers, sunset 2026-03-31). Tenant scoping is via the validated stellaops:tenant claim; gateway-forwarded tenant headers are envelope context, not the Notifier data-isolation source.

• List failed deliveries: GET /api/v2/notify/deliveries?status=failed&since=<utc>&limit=<n> (returns { items, count, continuationToken }; default limit 50).

• Retry a delivery: POST /api/v2/notify/deliveries/{deliveryId}/retry (optional body { reason, forceChannel }; rejects already Sent/Delivered with delivery_already_completed; appends a new attempt and audits delivery.retry).

• Delivery stats: GET /api/v2/notify/deliveries/stats.

• Dead-letter administration: GET /api/v2/notify/dead-letter, GET /api/v2/notify/dead-letter/{entryId}, GET /api/v2/notify/dead-letter/stats, POST /api/v2/notify/dead-letter/retry.

• Rotate a webhook secret: POST /api/v2/notify/security/webhook/{channelId}/rotate (returns new secret, activeAt, oldSecretExpiresAt); inspect masked value via GET /api/v2/notify/security/webhook/{channelId}/secret.

• Acknowledge an incident/delivery: POST /api/v2/notify/incidents/{incidentId}/ack (requires notify.operator; an actor/comment body is optional) or GET /api/v1/ack/{token} (token-signed public ack link; POST variant also exists for an optional comment/metadata body). A paired POST /api/v2/notify/incidents/{incidentId}/resolve (also notify.operator) closes the incident. A simplified surface at /api/v2/incidents/{deliveryId}/ack (notify.operator) exists alongside the /api/v2/notify/... paths.

• Event ingest (producers): producers publish events to Notifier via versioned v1 routes — POST /api/v1/notify/attestation-events, POST /api/v1/notify/risk-events, POST /api/v1/notify/pack-approvals (+ /{packId}/ack), and the producer-polymorphic POST /api/v1/events/{producer} (whitelisted producers: concelier, attestor, risk; requires notify.operator; the event kind must carry the producer’s prefix and the body tenant must equal the validated stellaops:tenant claim; timestamps must be within ±5m of server clock). These accept envelopes and enqueue onto notify:events; they return 202 Accepted. Note these v1 routes also emit Deprecation/Sunset headers.

NOT IMPLEMENTED as described in earlier drafts: there is no :replay delivery verb (use /retry), no channels/{id}:refresh-secret (use security/webhook/{id}/rotate), and no per-tenant pause admin flag in the WebService surface.

Failure recovery

• Worker crash loop: check Valkey/Redis (queue) connectivity and PostgreSQL reachability; inspect template compile/render errors. The Worker fails fast if the queue transport is unavailable (no NullNotifyEventQueue fallback outside the Testing environment).
• PostgreSQL outage: dispatch pauses; on recovery the queue redrives. Failed deliveries follow the retry policy and otherwise land in the dead-letter queue for inspection, alerting, and reprocessing (/dead-letter/retry).
• Channel outage (e.g., Slack/PagerDuty 5xx): the dispatcher retries transient errors with exponential backoff. As implemented, the webhook dispatcher attempts up to 3 times (WebhookChannelDispatcher, maxAttempts = 3) with backoff 2^(attempt-1) seconds (1s, then 2s), and the shared ChannelAdapterOptions defaults are MaxRetries = 3, base delay 1s, max delay 30s, dispatch timeout 30s (circuit breaker opens after 5 consecutive failures for 1m). HTTP 5xx (500/502/503/504) and network/timeout errors retry; HTTP 429 is treated as retryable and the channel adapters (Webhook/PagerDuty/ OpsGenie/Chat) parse a Retry-After header; other HTTP 4xx are permanent failures (no retry). Exhausted deliveries move to the dead-letter queue. For extended outages, disable the channel or route to a fallback. NOTE: the companion design doc docs/notifications/operations/retries.md describes an aspirational policy (initial 5s, 1h cap, up to 10 attempts over 24h) that does NOT match the current code — trust the values above.

Auditing

• Operations are written to the notify audit repository (e.g., delivery.retry, webhook.secret.rotated) with actor (X-StellaOps-Actor, default api), tenant, entity id/type, and UTC timestamp.
• Ack events resolve their external id (PagerDuty dedup_key / OpsGenie alias) against the first-class indexed notify.deliveries.external_id column; there is no JSON metadata fallback in production.
• Browse delivery history via GET /api/v2/notify/deliveries (JSON). NOTE: an NDJSON bulk export (?format=ndjson) is NOT implemented on this endpoint.

Determinism safeguards

• Deterministic delivery and stable ordering for notifications (module engineering rule).
• Template rendering enforces redaction defaults (safe / paranoid / none modes, field allowlists) and avoids secrets in logs; no live lookups during render.
• Time sources are UTC; quiet-hours/maintenance windows evaluated from durable tenant schedules (cron expressions evaluated natively, including after restart).
• Shared Notify persistence auto-converges the notify schema on startup; migrations are idempotent and forward-only.

On-call checklist

• [ ] WebService /healthz and Worker health endpoints green; queue health check passing.
• [ ] Delivery failure rate within SLO (NotifyDeliverySuccessSLO ≥ 0.99 over 5m).
• [ ] Backlog within SLO (NotifyBacklogDepth < 10000) and escalation backlog clear.
• [ ] P99 delivery latency within SLO (NotifyLatencyP99 < 5s).
• [ ] Valkey/Redis healthy and PostgreSQL primary healthy.
• [ ] Latest release notes applied and channels validated; egress allowlist current.

Related Documentation

• Notifier Architecture
• Notify Architecture
• Notify retry policy (design) — see the caveat under Failure recovery: the shipped retry behaviour differs from this design doc.
• SLO alert rules: docs/notifications/operations/alerts/notify-slo-alerts.yaml (see the Monitor caveat on alert-rule vs. emitted-instrument names).