Notify Runbook

Last updated: 2026-09-08 (NTF-10 — the Notifier predecessor retired; this runbook now targets the consolidated hosts). The filename is unchanged so existing operator bookmarks resolve.

Purpose

Operational steps to deploy, monitor, and recover the Notify service — the Notifications Studio. It owns the PostgreSQL stellaops_notify database and the Valkey/Redis (or NATS JetStream) notification queue.

Audience: operators and on-call engineers who deploy, monitor, and troubleshoot Notify in a Stella Ops deployment.

• notify-web(src/Notify/StellaOps.Notify.WebService, slot 29): the management/admin REST API — rule and template CRUD, simulation, delivery browsing, dead-letter administration, storm-breaker status, webhook secret rotation — plus event ingest.
• notify-worker(src/Notify/StellaOps.Notify.Worker, composing src/Notify/__Libraries/StellaOps.Notify.Delivery): queue-driven dispatch, channel delivery, rule evaluation, correlation, storm-breaking (dedup/throttle), escalation, digest, dead-letter handling, and retention cleanup.

Predecessor retired. notifier-web merged into notify-web in 2026-04 and was commented out of compose from then on; the notifier-worker deployable was retired on 2026-09-04 after a constructed forcing function proved notify-worker delivering, and the whole src/Notifier/ tree was frozen at src/__Obsoleted/Notifier/ on 2026-09-08 (SPRINT_20260722_015 NTF-10). stellaops/notifier-web and stellaops/notifier-worker are no longer built; notifier.stella-ops.local survives only as a DNS alias on the notify-web container. (The legacy split-container monolith compose was deleted 2026-09-01, SPRINT_20260824_010 P8-12.)

Pre-flight

• Authority client registered for Notify with the appropriate scopes: notify.viewer (read), notify.operator (rule/channel/delivery operations), notify.admin (secrets, escalations, platform settings), notify.escalate. (Defined in StellaOps.Auth.Abstractions/StellaOpsScopes.cs.)
• Channel credentials (SMTP, Slack/Teams/PagerDuty/OpsGenie, webhook HMAC keys) are stored as per-channel rows in the notify schema, not as service env vars. The dev-only SMTP override is the exception for local capture (Mailpit): on notify-worker the keys are Notifier__Email__Smtp__* with no env prefix, binding the unchanged Notifier:Email:Smtp section (DefaultSmtpOptions.SectionName). See ../../modules/notify/operations/dev-smtp.md.
• Outbound egress controlled by the AirGap egress policy (AddAirGapEgressPolicy); ensure target channel hosts are permitted under the active egress allowlist. (Inbound webhook IP allowlists are a separate per-channel control.)
• PostgreSQL and Valkey/Redis reachable; the notify schema auto-migrates on startup (forward-only; see ADR-004). Queue health check (AddNotifyQueueHealthCheck) passes.
• Offline kit loaded: channel manifests, default templates (devops/offline/notifier/templates/; there is no repo-root offline/ directory — the build-time copy that used to sit inside the predecessor host went with it at NTF-10), rule seeds. Clean default startup does NOT seed demo channels/templates/rules/incidents: the shared Notify persistence registration defaults runSeedMigrations to false. NOTIFY_BOOTSTRAP_ENABLED=true is the toggle that opts demo seeding back in, and it is read by notify-web.
• notify-web plug-in readiness requires the base Notify bundle mounted under notify:plugins:directory. The required base bundle contains only StellaOps.Notify.Connectors.InApp and StellaOps.Notify.Connectors.InAppInbox; optional email/chat/paging/regulatory connector overlays do not belong in the base stack.
• The notify-web container reads the runtime config from /app/etc/notify/notify.yaml; compose mounts devops/etc/notify/ read-only at /app/etc/notify, so place the selected notify.*.yaml template there as notify.yaml before startup.

Deploy

1. Apply the supported Compose or Offline Kit profile with image digests pinned. The images are stellaops/notify-web and stellaops/notify-worker. Both are declared in devops/docker/services-matrix.env and devops/docker/build-service-publish.sh under slot 29; slot 28 carries only a do-not-re-enable note for the retired predecessor.
2. Set config (__ maps to config section nesting; neither host applies an env-var prefix):
   • STELLAOPS_POSTGRES_NOTIFY_CONNECTION — the family’s own database. Both hosts resolve this variable and only this one, and fail closed without it (CoC §8.2).
   • Queue transport and connection under the Notify__Queue__* section.
   • Authority/Router auth wiring via the standard Router / Authority sections.
   • ASPNETCORE_URLS for the listener (default container port 8080).
3. Verify liveness/readiness: WebService GET /healthz returns 200 Healthy (ASP.NET health-check format, not a JSON ready flag). For the companion notify-web host, GET /readyz returns 503 with missing base connector names when the InApp/InAppInbox base plug-ins are absent or mounted at the wrong notify:plugins:directory. The Worker exposes its own worker health endpoints via MapWorkerHealthEndpoints (includes the queue health check).

NOT IMPLEMENTED: there is no warm-cache admin endpoint (POST .../admin/warm); rules and templates are loaded lazily/per-request and from durable storage.

Monitor

The delivery host emits OpenTelemetry instruments via the StellaOps.Notifier meter — the METER name is unchanged by the host retirement, because renaming it would break every existing dashboard and alert rule. Instrument names (dot-notation as emitted; a Prometheus exporter renders these with _/_total suffix conventions):

• Delivery: notify.delivery.attempts (counter, tags: tenant/channel/status), notify.delivery.duration (histogram, ms). Notifier-summary family also emits notifier.delivery.attempts / .successes / .failures and notifier.delivery.latency.
• Escalation: notify.escalation.events; notifier.escalation.ack_latency.
• Rule evaluation: notify.rule.evaluations, notify.rule.evaluation.duration.
• Templates: notify.template.renders, notify.template.render.duration.
• Storm/dedup: notify.storm.events; notifier.storm.events, notifier.suppression.count.
• Dead-letter: notify.deadletter.entries; notifier.deadletter.count.
• Queue depth: notify.queue.depth (observable gauge, per channel); retention via notify.retention.cleanups.

SLO alerting rules ship in docs/notifications/operations/alerts/notify-slo-alerts.yaml and reference the exporter metric names notify_delivery_success_total, notify_delivery_total, notify_backlog_depth, and notify_delivery_duration_seconds_bucket. CAVEAT: those alert-rule names do NOT map 1:1 onto the instruments the worker actually emits — the code emits notify.delivery.attempts (tagged by status, not a separate success counter), notify.delivery.duration, and notify.queue.depth (per-channel gauge), with no instrument literally named notify.delivery.success, notify.delivery (total), or notify.backlog.depth. Wiring the alerts to live data therefore requires recording rules / relabeling that derive the alert-rule series from the emitted instruments; the shipped rules are not satisfiable directly against the raw exporter output.

• Logs: structured console JSON with UTC timestamps; preserve/propagate X-StellaOps-Correlation-Id (GUID/ULID) across inbound and outbound calls.
• Traces: span notify.delivery (and notify.escalation) from ActivitySource StellaOps.Notifier.

Common operations

All management routes are under /api/v2/notify (v1 routes are deprecated and emit Deprecation/Sunset headers, sunset 2026-03-31). Tenant scoping is via the validated stellaops:tenant claim; gateway-forwarded tenant headers are envelope context, not the Notify data-isolation source.

• List failed deliveries: GET /api/v2/notify/deliveries?status=failed&since=<utc>&limit=<n> (returns { items, count, continuationToken }; default limit 50).

• Retry a delivery: POST /api/v2/notify/deliveries/{deliveryId}/retry (optional body { reason, forceChannel }; rejects already Sent/Delivered with delivery_already_completed; appends a new attempt and audits delivery.retry).

• Delivery stats: GET /api/v2/notify/deliveries/stats.

• Dead-letter administration: GET /api/v2/notify/dead-letter, GET /api/v2/notify/dead-letter/{entryId}, GET /api/v2/notify/dead-letter/stats, POST /api/v2/notify/dead-letter/retry.

• Rotate a webhook secret: POST /api/v2/notify/security/webhook/{channelId}/rotate (returns new secret, activeAt, oldSecretExpiresAt); inspect masked value via GET /api/v2/notify/security/webhook/{channelId}/secret.

• Acknowledge an incident/delivery: POST /api/v2/notify/incidents/{incidentId}/ack (requires notify.operator; an actor/comment body is optional) or GET /api/v1/ack/{token} (token-signed public ack link; POST variant also exists for an optional comment/metadata body). A paired POST /api/v2/notify/incidents/{incidentId}/resolve (also notify.operator) closes the incident. A simplified surface at /api/v2/incidents/{deliveryId}/ack (notify.operator) exists alongside the /api/v2/notify/... paths.

• Event ingest (producers): producers publish events to Notify via versioned v1 routes — POST /api/v1/notify/attestation-events, POST /api/v1/notify/risk-events, POST /api/v1/notify/pack-approvals (+ /{packId}/ack), and the producer-polymorphic POST /api/v1/events/{producer} (whitelisted producers: concelier, attestor, risk; requires notify.operator; the event kind must carry the producer’s prefix and the body tenant must equal the validated stellaops:tenant claim; timestamps must be within ±5m of server clock). These accept envelopes and enqueue onto notify:events; they return 202 Accepted. Note these v1 routes also emit Deprecation/Sunset headers.

NOT IMPLEMENTED as described in earlier drafts: there is no :replay delivery verb (use /retry), no channels/{id}:refresh-secret (use security/webhook/{id}/rotate), and no per-tenant pause admin flag in the WebService surface.

Failure recovery

• Worker crash loop: check Valkey/Redis (queue) connectivity and PostgreSQL reachability; inspect template compile/render errors. The Worker fails fast if the queue transport is unavailable (no NullNotifyEventQueue fallback outside the Testing environment).
• PostgreSQL outage: dispatch pauses; on recovery the queue redrives. Failed deliveries follow the retry policy and otherwise land in the dead-letter queue for inspection, alerting, and reprocessing (/dead-letter/retry).
• Channel outage (e.g., Slack/PagerDuty 5xx): the dispatcher retries transient errors with exponential backoff. As implemented, the webhook dispatcher attempts up to 3 times (WebhookChannelDispatcher, maxAttempts = 3) with backoff 2^(attempt-1) seconds (1s, then 2s), and the shared ChannelAdapterOptions defaults are MaxRetries = 3, base delay 1s, max delay 30s, dispatch timeout 30s (circuit breaker opens after 5 consecutive failures for 1m). HTTP 5xx (500/502/503/504) and network/timeout errors retry; HTTP 429 is treated as retryable and the channel adapters (Webhook/PagerDuty/ OpsGenie/Chat) parse a Retry-After header; other HTTP 4xx are permanent failures (no retry). Exhausted deliveries move to the dead-letter queue. For extended outages, disable the channel or route to a fallback. NOTE: the companion design doc docs/notifications/operations/retries.md describes an aspirational policy (initial 5s, 1h cap, up to 10 attempts over 24h) that does NOT match the current code — trust the values above.

Auditing

• Operations are written to the notify audit repository (e.g., delivery.retry, webhook.secret.rotated) with actor (X-StellaOps-Actor, default api), tenant, entity id/type, and UTC timestamp.
• Ack events resolve their external id (PagerDuty dedup_key / OpsGenie alias) against the first-class indexed notify.deliveries.external_id column; there is no JSON metadata fallback in production.
• Browse delivery history via GET /api/v2/notify/deliveries (JSON). NOTE: an NDJSON bulk export (?format=ndjson) is NOT implemented on this endpoint.

Determinism safeguards

• Deterministic delivery and stable ordering for notifications (module engineering rule).
• Template rendering enforces redaction defaults (safe / paranoid / none modes, field allowlists) and avoids secrets in logs; no live lookups during render.
• Time sources are UTC; quiet-hours/maintenance windows evaluated from durable tenant schedules (cron expressions evaluated natively, including after restart).
• Shared Notify persistence auto-converges the notify schema on startup; migrations are idempotent and forward-only.

On-call checklist

• [ ] WebService /healthz and Worker health endpoints green; queue health check passing.
• [ ] Delivery failure rate within SLO (NotifyDeliverySuccessSLO ≥ 0.99 over 5m).
• [ ] Backlog within SLO (NotifyBacklogDepth < 10000) and escalation backlog clear.
• [ ] P99 delivery latency within SLO (NotifyLatencyP99 < 5s).
• [ ] Valkey/Redis healthy and PostgreSQL primary healthy.
• [ ] Latest release notes applied and channels validated; egress allowlist current.

Related Documentation

• Notify Architecture
• Dev SMTP loop (Mailpit)
• Notifier freeze record — where the retired predecessor went
• Notify retry policy (design) — see the caveat under Failure recovery: the shipped retry behaviour differs from this design doc.
• SLO alert rules: docs/notifications/operations/alerts/notify-slo-alerts.yaml (see the Monitor caveat on alert-rule vs. emitted-instrument names).