Notifier (Notifications Studio Host)

Notifier is the deployable host for the Stella Ops Notifications Studio — the WebService and Worker that turn platform events into delivered, audited notifications. This page is for operators running the notifier and for engineers integrating with its event-ingest and management APIs.

Status: Implemented Source: src/Notifier/ Owner: Notify Guild

Note: Notifier is the deployment host for the Notifications Studio. For the underlying notification toolkit (engine, storage, queue, connectors), see ../notify/.

Purpose

Notifier provides the deployable WebService and Worker that compose the Notify libraries into the Notifications Studio experience. It is the entry point for event ingestion, rule/channel/template management, delivery history, incident correlation and escalation, on-call schedules, quiet-hours and maintenance suppression, throttling and storm-breaking, localization, dead-letter administration, retention, and webhook security.

The WebService and Worker are independent ASP.NET Core hosts (both target net10.0) that share the Notify PostgreSQL notify schema and the Notify event queue.

Latest Updates

2026-04-15: production Notifier runtime no longer shadows the shared Notify persistence/queue stack with in-memory repositories.
2026-04-15: pack-approval ingestion persists durably in notify.pack_approvals and uses durable notify.locks idempotency coordination.
2026-04-15: restart-survival proof now exists via NotifierDurableRuntimeProofTests, covering submit -> persist -> process -> readback on real Postgres + Redis.
2026-04-16: NullNotifyEventQueue is no longer available in Development; only the Testing environment may use the null queue fallback.
2026-04-16: non-testing throttle and operator-override admin APIs now persist through PostgreSQL-backed suppression services and legacy compat adapters; restart-survival proof is covered by NotifierSuppressionDurableRuntimeTests.
2026-04-16: non-testing escalation-policy and on-call schedule APIs now persist through PostgreSQL-backed services and legacy compat adapters; restart-survival proof is covered by NotifierEscalationOnCallDurableRuntimeTests.
2026-04-16: non-testing quiet-hours calendars and maintenance windows now persist through PostgreSQL-backed runtime services and legacy compat adapters; restart-survival proof is covered by NotifierQuietHoursMaintenanceDurableRuntimeTests.
2026-04-20: compat /api/v2/notify/quiet-hours/* schedules that persist non-projectable cron expressions now evaluate natively after restart, and canonical /api/v2/quiet-hours/calendars/* reads surface the original cronExpression and duration metadata instead of leaving those schedules as inert 00:00 placeholders.
2026-04-16: non-testing webhook security, tenant isolation, dead-letter administration, and retention cleanup state now persist through PostgreSQL-backed runtime services and legacy compat adapters; restart-survival proof is covered by NotifierSecurityDeadLetterDurableRuntimeTests.
2026-04-20: non-testing worker dispatch now composes adapter-backed Email, PagerDuty, and OpsGenie delivery channels alongside webhook/chat dispatch, and durable delivery persistence now preserves provider externalId plus incidentId metadata for restart-safe external acknowledgements.
2026-04-20: PagerDuty and OpsGenie inbound acknowledgement webhooks no longer depend on a process-local bridge map; restart-survival proof is covered by NotifierAckBridgeRuntimeDurableTests and live worker DI proof by NotifierWorkerHostWiringTests.
2026-04-20: legacy /api/v2/notify/simulate* endpoints now use the same DI-composed simulation runtime as /api/v2/simulate*, and maintenance-window suppression parity is covered by SimulationEndpointsBehaviorTests.LegacySingleEventSimulation_UsesQuietHoursEvaluator_ForMaintenanceSuppression.
2026-04-20: the live Notifier worker now composes scheduled digest services through AddDigestServices(...); DigestScheduleRunner resolves tenant IDs from schedule configuration instead of the in-memory tenant stub, and worker-host proof is covered by NotifierWorkerHostWiringTests.
2026-04-20: the orphaned cron-based IDigestScheduler path was retired from the worker. Scheduled digests are now documented as configuration-driven only, while /digests remains the admin surface for open digest windows rather than schedule CRUD.
2026-04-20: WebService startup now registers durable quiet-hours, suppression, escalation/on-call compat, security, and dead-letter services directly for non-testing hosts. The remaining in-memory variants are isolated to the Testing environment, and startup-contract proof lives in StartupDependencyWiringTests.
2026-04-24: non-testing WebService and Worker startup now use durable correlation, escalation, storm-breaker, localization, fallback, acknowledgement, digest, and security/dead-letter runtime services. Testing keeps explicit local harness registrations, and the process-local in-app channel adapter is gated out of the production worker; startup proof is covered by StartupDependencyWiringTests and NotifierWorkerHostWiringTests.
2026-04-24: shared Notify PostgreSQL persistence now registers startup migrations from both configuration and explicit-options composition paths, so Notifier hosts using either path auto-converge the notify schema on startup.
2026-04-29: ack-bridge resolver reads notify.deliveries.external_id as a first-class indexed column and no longer probes event_payload.metadata.externalId. Migration 009 backfills existing rows into columns and migration 010 prunes redundant metadata.externalId / metadata.channel JSON keys.
2026-07-11: the production delivery worker dispatcher map now includes the durable InAppInboxChannelAdapter; rules targeting InAppInbox persist inbox rows instead of failing with No dispatcher for InAppInbox. Delivered rows now persist as the terminal delivered storage state, preventing the pending-delivery poller from sending the same inbox notification repeatedly. Focused proof lives in InAppInboxChannelDispatcherTests, NotifyPersistenceModelMapperTests, NotifierDeliveryTerminalStateDurableRuntimeTests, and NotifierWorkerHostWiringTests.

Operations: ack-bridge external-ID durability

The PagerDuty / OpsGenie acknowledgement webhooks resolve their inbound externalId (PagerDuty dedup_key, OpsGenie alias) against notify.deliveries to find the matching tenant + incident. The resolver has one durable path: indexed lookup against notify.deliveries.external_id joined to notify.channels on (channel_id, tenant_id). It survives Postgres restarts and connection-pool drops because the value lives in a real column with the partial composite index idx_notify_deliveries_external_id (channel_id, external_id) WHERE external_id IS NOT NULL (added in migration 009_deliveries_external_id_columns.sql).

There is no JSON metadata fallback in non-testing code. Rows that still carry legacy event_payload.metadata.externalId / metadata.channel are normalized by migration 009_deliveries_external_id_columns.sql and cleaned by 010_deliveries_drop_metadata_externalid_keys.sql. If a delivery is missing the first-class external_id, the ack bridge does not resolve it from JSON.

Relationship to Notify

Component	Path	Purpose
Notify	`src/Notify/`	Reusable toolkit: engine, models, connectors, queue
Notifier	`src/Notifier/`	Host: WebService and Worker that compose Notify

Per 2025-11-02 module boundary decision: Maintain separation for packaging, offline kit parity, and cross-module governance.

Components

Deployables:

StellaOps.Notifier.WebService - REST API for event ingestion plus rules/channels/templates CRUD, delivery browsing/retry/stats, incident correlation and escalation, on-call schedules, quiet-hours/maintenance suppression, throttle configs, operator overrides, storm-breaker, localization bundles, in-app inbox, webhook/HTML/tenant security, dead-letter administration, retention, and OpenAPI. (Program.cs, Endpoints/*.cs)
StellaOps.Notifier.Worker - Event consumers (NotifierEventWorker), rule evaluator (DefaultNotifyRuleEvaluator), template renderers, channel dispatchers (AdapterChannelDispatcher, WebhookChannelDispatcher), delivery workers (DeliveryDispatchWorker, NotifierDispatchWorker), digest scheduling, escalation engine, and ack bridge. (Program.cs, Processing/, Dispatch/, Escalation/)

Integration Points:

Uses StellaOps.Notify.Models, StellaOps.Notify.Queue, StellaOps.Notify.Persistence, StellaOps.Notify.Engine.
Channel adapter types (NotifyChannelType, src/Notify/__Libraries/StellaOps.Notify.Models/NotifyEnums.cs): Slack, Teams, Email, Webhook, Custom, PagerDuty, OpsGenie, Cli, InAppInbox, InApp, Discord, Telegram. The worker registers dedicated adapters for Webhook, Email, PagerDuty, OpsGenie, InAppInbox, a chat-webhook adapter (ChatWebhookChannelAdapter, reported as Slack) used for Slack/Teams/Discord/Telegram-style chat sinks, and an in-app adapter (local dev/testing only). Note: Discord/Telegram persist as webhook in the notify.channel_type PostgreSQL enum.
Storage: PostgreSQL notify schema (auto-migrated via AddNotifyPersistence / AddStartupMigrations; migrations 001–012 plus S001_demo_seed under src/Notify/__Libraries/StellaOps.Notify.Persistence/Migrations/). Tables include channels, rules, templates, deliveries (partitioned), digests, quiet_hours, maintenance_windows, escalation_policies, escalation_states, on_call_schedules, inbox, incidents, audit, locks, and pack_approvals.
Queue: Redis (Valkey) Streams or NATS JetStream (selected via notifier:queue). Stream name notify:events. In the Testing environment a NullNotifyEventQueue fallback may be used.
Authentication: resource-server JWT validation against Authority (AddStellaOpsResourceServerAuthentication). Audit events are emitted to the Timeline service (AddAuditEmission).

Authorization scopes

The WebService validates the following scopes (defined canonically in StellaOps.Auth.Abstractions.StellaOpsScopes and mapped to named policies in WebService/Constants/NotifierPolicies.cs):

Scope	Policy constant	Purpose
`notify.viewer`	`NotifyViewer`	Read-only access to channels, rules, templates, delivery history, observability. Default policy on the `/api/v2/notify` group.
`notify.operator`	`NotifyOperator`	Rule/channel/template authoring, delivery actions, simulation, event ingest (`POST /api/v1/events/{producer}`).
`notify.admin`	`NotifyAdmin`	Security configuration, signing-key rotation, tenant-isolation grants, retention policies, platform-wide settings.
`notify.escalate`	`NotifyEscalate`	Escalation actions and on-call/escalation-policy management.

Most tenant-data routes require a validated stellaops:tenant claim; the gateway may still forward X-StellaOps-TenantId as envelope context, but Notifier does not use raw tenant headers as the data-isolation source. Write endpoints read an optional X-StellaOps-Actor header for audit attribution.

API surface

The route table below is grounded in WebService/Program.cs and WebService/Endpoints/*.cs. Some /api/v2/notify/* routes in Program.cs and the corresponding /api/v2/* group endpoints (e.g. /api/v2/notify/quiet-hours vs /api/v2/quiet-hours/calendars) coexist as canonical + legacy-compat surfaces; see the Latest Updates entries.
Authorization caveat (verify before relying on it): the /api/v2/notify rules/templates/incidents endpoints (Endpoints/NotifyApiEndpoints.cs) and the other Endpoints/*.cs groups (escalation, fallback, incidents, localization, observability, security) attach .RequireAuthorization(...) per route. The inline /api/v2/notify/* routes defined directly in Program.cs (channels, deliveries, deliveries/retry+stats, simulate, quiet-hours, maintenance-windows, throttle-configs, overrides, escalation-policies, oncall-schedules, inbox, localization, storms, security, dead-letter, retention) do not call .RequireAuthorization and rely on tenant-claim context middleware only — they are not scope-gated at the route level. Only POST /api/v1/events/{producer} among the Program.cs routes requires a scope (notify.operator). The PagerDuty/OpsGenie ack webhooks under /api/v2/ack/webhook/* are AllowAnonymous.

Event ingest (v1):

POST /api/v1/events/{producer} — producer-polymorphic ingest (concelier / attestor / risk); enforces tenant-claim/body match and kind-prefix. Requires notify.operator.
POST /api/v1/notify/pack-approvals, POST /api/v1/notify/pack-approvals/{packId}/ack — pack-approval ingest + acknowledgement (Idempotency-Key required).
POST /api/v1/notify/attestation-events, POST /api/v1/notify/risk-events — typed event ingest.
GET /api/v1/ack/{token}, POST /api/v1/ack/{token} — escalation acknowledgement via signed token.

Notify v2 (/api/v2/notify): Rules/templates/incidents are mapped by MapNotifyApiV2 with a notify.viewer group default and notify.operator on writes. The remaining routes in this section are mapped inline in Program.cs and are not scope-gated (see the authorization caveat above).

Rules: GET /rules, GET /rules/{ruleId}, POST /rules, PUT /rules/{ruleId}, DELETE /rules/{ruleId}.
Templates: GET /templates, GET /templates/{templateId}, POST /templates, DELETE /templates/{templateId}, POST /templates/preview, POST /templates/validate.
Channels: GET /channels, GET /channels/{channelId}, PUT /channels/{channelId}, DELETE /channels/{channelId}.
Deliveries: GET /deliveries, GET /deliveries/{deliveryId}, POST /deliveries/{deliveryId}/retry, GET /deliveries/stats.
Incidents: GET /incidents, POST /incidents/{incidentId}/ack, POST /incidents/{incidentId}/resolve.
Simulation: POST /simulate, POST /simulate/event.
Suppression: GET|PUT|DELETE /quiet-hours[/{scheduleId}], GET|PUT|DELETE /maintenance-windows[/{windowId}], GET|PUT|DELETE /throttle-configs[/{configId}], GET /overrides, POST /overrides, DELETE /overrides/{overrideId}.
Escalation: GET|PUT|DELETE /escalation-policies[/{policyId}], GET|PUT|DELETE /oncall-schedules[/{scheduleId}], POST /oncall-schedules/{scheduleId}/overrides, DELETE /oncall-schedules/{scheduleId}/overrides/{overrideId}.
Inbox: GET /inbox, GET /inbox/{messageId}, POST /inbox/{messageId}/read, POST /inbox/read-all, GET /inbox/unread-count, DELETE /inbox/{messageId}.
Localization: GET /localization/bundles[/{bundleId}], PUT|DELETE /localization/bundles/{bundleId}, GET /localization/locales, POST /localization/resolve.
Storms: GET /storms, POST /storms/{stormKey}/summary.
Security: POST /security/ack-tokens, POST /security/ack-tokens/verify, POST /security/html/validate, POST /security/html/sanitize, POST /security/webhook/{channelId}/rotate, GET /security/webhook/{channelId}/secret, GET /security/isolation/violations.
Dead-letter: POST|GET /dead-letter, GET /dead-letter/{entryId}, POST /dead-letter/retry, POST /dead-letter/{entryId}/resolve, GET /dead-letter/stats, POST /dead-letter/purge.
Retention: GET|PUT /retention/policy, POST /retention/cleanup, GET /retention/cleanup/preview, GET /retention/cleanup/last.

Other v2 route groups (registered from Endpoints/*.cs): /api/v2/rules, /api/v2/templates, /api/v2/incidents (+ WebSocket live feed via MapIncidentLiveFeed), /api/v2/simulate, /api/v2/quiet-hours, /api/v2/throttles, /api/v2/overrides, /api/v2/escalation-policies, /api/v2/oncall-schedules, /api/v2/escalations, /api/v2/ack (incl. /webhook/pagerduty, /webhook/opsgenie), /api/v2/storm-breaker, /api/v2/localization, /api/v2/fallback, /api/v2/security, and /api/v1/observability (metrics, dead-letters, chaos experiments, retention policies).

Operational: GET /healthz (health check), GET /.well-known/openapi (OpenAPI document). Requests under /api/v1 emit RFC 8594 Deprecation/Sunset headers.

Notify architecture — the underlying toolkit Notifier hosts.
Notifier dev SMTP defaults.
Authority — OAuth clients and scopes.
JobEngine — Scheduler event sources (Concelier/Excititor export webhooks).