Notifier — Architecture (Notifications Studio Host)

Status: Implemented Source: src/Notifier/ Owner: Notify Guild Scope. Architecture dossier for Notifier, the src/Notifier/ tree that composes the reusable Notify toolkit into runtime services. This document covers how Notifier wires the Notify libraries and where the persistence and queue boundaries sit. For the toolkit internals (rules engine semantics, event model, template engine, schema design), read the companion dossier: ../notify/architecture.md.

⚠ Deployed topology (corrected 2026-07-12, SPRINT_20260712_009 CLO-7). Of the two projects in src/Notifier/, only StellaOps.Notifier.Worker (notifier-worker) is deployed. StellaOps.Notifier.WebService is commented out of compose — # Slot 28: Notifier (MERGED into notify-web — kept commented for rollback) (devops/compose/docker-compose.stella-services.yml) — so it is not an operator-deployable host. The full Notifications Studio REST surface (v1 + the complete v2 enterprise route set) is served by notify-web(src/Notify/StellaOps.Notify.WebService, Program.cs:453-468). The Notifier WebService is a divergent duplicate of that surface retained solely as a rollback path; the KEEP decision is deliberate (audit corrections §A2) but the code must not be read as the live API host. Sections below that describe the WebService describe the rollback copy, not the deployed API. This dossier’s central “Notifier is the deployable host” framing predates the merge and is being reconciled.

1) Purpose & scope

Notifier owns the runtime composition that turns the Notify libraries into running services. The src/Notifier/ tree contains two projects:

• StellaOps.Notifier.Worker— the deployed service (notifier-worker): the background pipeline — event consumers, rule evaluation, template rendering, and channel delivery, plus correlation/escalation/storm-breaker/digest runtimes.
• StellaOps.Notifier.WebService— a REST API for the Studio (rules/channels CRUD, test send, delivery browsing, and the enterprise admin surface). Not deployed — commented out of compose and retained only as a rollback copy (see the banner above); the live equivalent runs in notify-web (src/Notify/).

Both compose libraries that live in src/Notify/through dependency injection. The split between the two trees is intentional:

This boundary is the 2025-11-02 module boundary decision (recorded in ../notify/architecture.md§0 and in README.md): keep src/Notify/ as the reusable toolkit and src/Notifier/ as the host that composes it, and do not merge the directories without an approved packaging RFC covering build impacts, Offline Kit parity, and cross-module governance.

What Notifier does not own

• It does not implement the rules engine, idempotency, throttling, or template rendering primitives — those are in StellaOps.Notify.Engine.
• It does not define DTOs/contracts — those are in StellaOps.Notify.Models.
• It does not own the channel connector implementations — those are the StellaOps.Notify.Connectors.* plug-ins.
• It does not define the notify PostgreSQL schema or its migrations — those are owned by StellaOps.Notify.Persistence. Notifier hosts that schema (it runs the startup migrations), but the schema authority is the toolkit. See §4.

2) Service composition (WebService + Worker)

Both deployables are .NET minimal-host ASP.NET Core apps. They bind configuration from appsettings.json plus environment variables under the NOTIFIER_ prefix, then compose the Notify libraries via DI extension methods.

2.1 WebService

StellaOps.Notifier.WebService (entry: Program.cs) wires, in order:

• Persistence — AddNotifyPersistence(configuration, "notifier:storage:postgres") (from StellaOps.Notify.Persistence). This registers the canonical Postgres repositories and the notify-schema startup migration host.
• Correlation/suppression runtimes — durable, PostgreSQL-backed services in non-testing environments (AddDurableCorrelationServices); in-memory equivalents are gated to the Testing environment only.
• Rule evaluation + simulation — INotifyRuleEvaluator (DefaultNotifyRuleEvaluator) and the simulation services that back the /api/v2/simulate* endpoints.
• Event queue — AddNotifyEventQueue(configuration, "notifier:queue") from StellaOps.Notify.Queue. The NullNotifyEventQueue fallback is allowed only in the Testing environment.
• Admin runtime services — quiet hours, maintenance windows, throttle configuration, operator overrides, escalation/on-call, webhook security, dead-letter, and retention, each resolving to PostgreSQL-backed implementations in non-testing hosts and in-memory variants under Testing.
• Templates — AddTemplateServices(...) with the worker-side enhanced renderer.
• Transport + auth — Router AspNet messaging transport (StellaOps.Router.AspNet, StellaOps.Router.Transport.Messaging) and Authority server integration (StellaOps.Auth.ServerIntegration, via AddStellaOpsResourceServerAuthentication) for OpTok/tenant enforcement. The WebService registers four named scope policies (AddStellaOpsScopePolicy) backed by the canonical scope catalog (StellaOps.Auth.Abstractions.StellaOpsScopes): notify.viewer (read-only), notify.operator (rule/channel/template/delivery management + simulation), notify.admin (security config, signing-key rotation, tenant-isolation grants, retention), and notify.escalate (incident start/escalate/stop, escalation policies, on-call). The constants live in NotifierPolicies.

The WebService also references the Worker project directly, because several admin/runtime service contracts (channels, security, storm-breaker, dead-letter, retention, observability, escalation, tenancy, templates, correlation) are defined in the Worker assembly and reused by the API surface.

The HTTP surface combines endpoint-class registrations (MapRuleEndpoints, MapTemplateEndpoints, MapIncidentEndpoints, MapSimulationEndpoints, MapQuietHoursEndpoints, MapThrottleEndpoints, MapOperatorOverrideEndpoints, MapEscalationEndpoints, MapStormBreakerEndpoints, MapLocalizationEndpoints, MapFallbackEndpoints, MapSecurityEndpoints, MapObservabilityEndpoints) under /api/v2/{resource} (e.g. /api/v2/rules, /api/v2/templates, /api/v2/simulate, /api/v2/incidents, /api/v2/escalations, /api/v2/quiet-hours, /api/v2/throttles, /api/v2/overrides, /api/v2/security, /api/v2/storm-breaker, /api/v2/localization, /api/v2/ack, /api/v2/fallback, /api/v1/observability) with inline-mapped routes in Program.cs for channels/deliveries (/api/v2/notify/channels, /api/v2/notify/deliveries) and the event-ingress surface (/api/v1/notify/*, /api/v1/events/{producer}). It also wires a WebSocket live incident feed: UseWebSockets(...) plus MapIncidentLiveFeed() at /api/v2/incidents/live (header- or query-tenant-scoped subscriptions).

Event ingress. Producers push events over HTTP as well as the queue. Typed v1 routes exist for pack approvals (/api/v1/notify/pack-approvals, with an /{packId}/ack companion), attestation events (/api/v1/notify/attestation-events), and risk events (/api/v1/notify/risk-events). A single producer-polymorphic route POST /api/v1/events/{producer} (Sprint 20260513_002 G1) accepts envelopes from a whitelisted producer set — concelier, attestor, risk — enforcing that the event kind carries the producer’s prefix (concelier. / attestor. / risk.) and that the body tenant matches the X-StellaOps-TenantId header; it is gated by notify.operator and tagged Events. All /api/v1/* routes emit RFC 8594 Deprecation/Sunset headers (Sunset 31 Mar 2026). Ingest routes enqueue a NotifyEvent onto the notify:events stream for the Worker to dequeue.

Environment gating is a first-class contract. Direct in-memory runtime registrations exist only for the Testing environment; non-testing hosts resolve PostgreSQL-backed runtime services and a real Redis/NATS queue. Startup-contract proof for this gating lives in the Notifier test suites (e.g. StartupDependencyWiringTests).

2.2 Worker

StellaOps.Notifier.Worker (entry: Program.cs, built via CreateSlimBuilder) composes:

• Persistence + queue — the same AddNotifyPersistence(...) / AddNotifyEventQueue(...) pair, so WebService and Worker converge on identical storage and queue bindings.
• Durable runtime services — AddDurableCorrelationServices, AddDurableEscalationServices, AddDurableStormBreakerServices, and AddDigestServices(...) (with ConfiguredDigestTenantProvider resolving tenant IDs from configuration in non-testing hosts; testing keeps the non-durable variants).
• Event processing — NotifierEventProcessor plus the hosted NotifierEventWorker that leases events from the queue; event handlers such as FederationBundleEventHandler fan specific producer events (e.g. concelier.federation.bundle.ready / .failed) out to inbox + email.
• Operator provider-change fan-out (OSK-P5, 2026-07-19) — the generic producer endpoint admits platform.* only for the platform producer. CryptoProviderChangeEventHandler consumes the exact platform.crypto-provider-changed kind. Before rendering, CryptoProviderChangeNoiseGate applies maintenance/static quiet-hours through the correlation evaluator, checks persisted tenant quiet-hours calendars, and obtains one atomic tenant+kind throttle lease. The stable event id owns that lease, so a retry may finish partial recipient fan-out while a distinct rapid provider change is suppressed across worker replicas. The handler then de-duplicates each event/subject pair in the durable lock repository, renders tenant-overridable templates with bootstrap fallback, dispatches the in-app leg through InAppInboxChannelDispatcher, and optionally uses the existing Email adapter path when the Authority-resolved recipient carries an address. CryptoProviderChangeTemplateSeeder upserts the built-in inbox/email templates at WebService startup; both include /administration/profile.
• Channel dispatch — AddChannelAdapters(...) (the process-local in-app adapter is gated to testing/development), plus the deterministic dispatch split: WebhookChannelDispatcher for chat/webhook routes and AdapterChannelDispatcher for adapter-backed channels (Email, PagerDuty, OpsGenie). A purpose-specific DoraInfoSharingRuntimeDispatcher handles dora-info-sharing channels. The hosted DeliveryDispatchWorker drives delivery.
• Air-gap egress policy — AddAirGapEgressPolicy(...) so outbound channel calls honour sealed-mode rules.
• Health — worker health checks plus the Notify queue health check.
• Plugin probe evidence - NotifierWorkerPluginProbeReportWriter writes the canonical worker report to /var/lib/stellaops/plugin-scratch/notifier/probe-report.json by default, using StellaOps:Plugins:ScratchRoot when configured. The DORA info-sharing row is driven by the mounted connector runtime. Optional DORA rows are omitted when optional plugins are not requested; an explicitly requested or profile-enabled DORA row is blocking. Missing bundles report mounted=false / isAccepted=false; unsigned or bad-signature bundles report mounted=true / status=rejected; and an admitted mounted implementation assembly can report mounted=true, admitted=true, loaded=true, and probed=true in dry-run evidence.

3) Data flow

The host wires a single linear pipeline; the logic of each stage lives in the Notify toolkit.

                         ┌──────────────────────────────────────────────┐
 platform events         │  StellaOps.Notifier.Worker (host)             │
 (Scanner, Scheduler,    │                                              │
  VEX Lens, Attestor,    │   NotifierEventWorker  ── leases ──► Queue   │
  Concelier, Zastava …)  │        │                  (Notify.Queue)     │
        │                │        ▼                                     │
        └── enqueue ────►│   NotifierEventProcessor                     │
                         │        │  rule eval (Notify.Engine)          │
                         │        │  throttle / digest / quiet hours    │
                         │        ▼                                     │
                         │   template render (Notify.Engine + Templates)│
                         │        │                                     │
                         │        ▼                                     │
                         │   dispatch:                                  │
                         │     WebhookChannelDispatcher (chat/webhook)  │
                         │     AdapterChannelDispatcher (Email/PD/OG)   │
                         │     DoraInfoSharingRuntimeDispatcher (DORA)  │
                         └────────────────┬─────────────────────────────┘
                                          │ persist delivery state
                                          ▼
                         notify PostgreSQL schema (Notify.Persistence)
                                          ▲
 admin / browse ◄────────────────────────┘
 (StellaOps.Notifier.WebService: rules, channels, deliveries, simulation,
  quiet hours, throttles, escalation, on-call, dead-letter, retention)

1. Ingress. Platform events arrive on the internal event bus and are leased by the hosted NotifierEventWorker from the queue transport configured under notifier:queue.
2. Evaluation. NotifierEventProcessor applies tenant-scoped rules through the Notify engine (INotifyRuleEvaluator / DefaultNotifyRuleEvaluator), then noise controls — throttles, digest coalescing, quiet-hours/maintenance suppression. The semantics of matchers, idempotency keys, and digest windows are defined by the toolkit (see ../notify/architecture.md§4 and ../notify/rules.md).
3. Render. Matched actions are rendered to channel-specific payloads via the template services.
4. Delivery. Rendered messages are dispatched through the deterministic dispatcher split and persisted as delivery rows.
5. Admin / browse. The WebService exposes CRUD and read surfaces over the same persisted state, including delivery history, simulation dry-runs, and the enterprise admin endpoints.

Inbound acknowledgements (PagerDuty dedup_key, OpsGenie alias) resolve against notify.deliveries.external_id — a first-class indexed column — so external acks survive restarts and pool drops (see README.md“ack-bridge external-ID durability”).

3.1 In-app inbox alert grouping (re-notification cap)

A repeating platform signal (e.g. deployment.drift re-firing every 30–60 s per drifted component) must not create one inbox row per fire — that once let unreadCount climb to ~5,300 and saturated the bell badge. The in-app inbox (notify.inbox) therefore collapses repeats server-side:

• Group key. NotifierEventProcessor threads the authoritative event scope (component/repo/image/namespace/digest) and source correlation onto delivery metadata. InAppInboxChannelDispatcher builds v2:{lower(kind)}:sha256:{sha256(lower(kind) || 0x1f || lower(subject))} through the central ICryptoHash binding with a fixed SHA-256 algorithm. Only typed resource scope can supply the subject; rendered title/body, generic subject, and action/schedule digest metadata never create a group. An event without stable resource identity stays ungrouped.
• Upsert and retry identity. IInboxRepository.AppendOrGroupAsync performs an active-group upsert, advances last_seen_at, refreshes the latest payload, and increments occurrence_count only when last_occurrence_id differs. The partial unique index uq_inbox_active_group is the conflict target, while delivery kind and source correlation remain distinct from presentation category and persistence row id.
• Bounded recurrence. Read state is sticky only while the same incident continues inside the 15-minute recurrence window. A gap greater than 15 minutes archives the prior active row; a recurring signal starts a fresh unread row with occurrence count 1. Explicit archive also frees the active slot.
• Read model. GET /api/v2/notify/inbox returns occurrenceCount, lastSeenAt, and groupKey per row; unreadCount counts distinct unacknowledged groups. The console flyout and /notifications page render the true rolled-up count (“×N · last seen …”).
• Migration 003_sprint20260717_notifier_inbox_grouping.sql is immutable legacy release history because its checksum may already be recorded at customer sites. Forward-only 004_sprint20260720_secure_bounded_inbox_grouping.sql adds last_occurrence_id, upgrades recoverable typed subjects to the identical v2 UTF-8/SHA-256 formula, retires unverifiable legacy keys without discarding their counters, and converges indexes/session state idempotently. PostgreSQL proof covers fresh startup, reapplication, and upgrade from a ledger/schema where 003 was already applied. Rows that legacy 003 had already deleted cannot be reconstructed and are not claimed as recovered evidence.

4) Persistence & queue wiring

4.1 Persistence — schema ownership boundary

The notify PostgreSQL schema is owned by the toolkit (StellaOps.Notify.Persistence) and hosted by Notifier:

• The schema definition and forward-only migrations are embedded SQL resources inside StellaOps.Notify.Persistence (Migrations/**/*.sql is declared <EmbeddedResource> in the .csproj). The live embedded set is 001_v1_notify_baseline.sql (the collapsed pre-1.0 001–012 chain), 002_sprint040_escalation_external_id_indexes.sql, immutable legacy grouping migration 003_sprint20260717_notifier_inbox_grouping.sql, forward-only secure repair 004_sprint20260720_secure_bounded_inbox_grouping.sql, and seed S001_v1_notify_seed_baseline.sql; the old numbered files sit under Migrations/_archived/pre_1.0/mig061/ and are excluded from embedding (.csproj Exclude="Migrations\_archived\**\*.sql") — cite the baseline, not the archived numbers.
• Both Notifier hosts call AddNotifyPersistence(...), which registers AddStartupMigrations(schemaName: "notify", moduleName: "Notify", migrationsAssembly: <Notify.Persistence assembly>). This satisfies the platform’s auto-migration requirement: any Notifier host converges the notify schema on startup from embedded resources, on a fresh database, without manual psql or external init scripts.
• Both composition paths register migrations — AddNotifyPersistence(IConfiguration, sectionName) and the explicit-options AddNotifyPersistence(Action<PostgresOptions>) — so a host wiring either path still auto-migrates.
• Notifier binds the Postgres options from configuration section notifier:storage:postgres. The repositories (channels, deliveries, rules, templates, digests, quiet hours, maintenance windows, escalation policies/state, on-call, inbox, incidents, audit, locks, throttle config, operator overrides, localization bundles, pack approvals, incident-report timeline state, anomaly subscriptions) are all registered by the toolkit extension; Notifier’s Worker adds durable adapter registrations (AddDurableNotifyWorkerStorage) that bridge the worker-side service contracts onto these repositories.

The take-away: Notify owns persistence; Notifier hosts it. Notifier never defines tables — it points the toolkit at a connection string and runs the toolkit’s migrations.

Durable state is the production default. The legacy in-memory repositories are isolated to the Testing environment; restart-survival of submit → persist → process → readback against real Postgres + Redis is proven by the Notifier durable-runtime test suites (e.g. NotifierDurableRuntimeProofTests, plus the suppression/escalation/quiet-hours/security dead-letter durable variants).

4.2 Queue binding

The event queue is bound by AddNotifyEventQueue(configuration, "notifier:queue") from StellaOps.Notify.Queue. The transport is selected from configuration via NotifyQueueTransportKind:

• Redis→ RedisNotifyEventQueue (Valkey/Redis Streams).
• Nats→ NatsNotifyEventQueue (NATS JetStream).

When required configuration is missing the registration falls back to UnavailableNotifyEventQueue, which fails closed rather than silently dropping events; the NullNotifyEventQueue no-op fallback is permitted only in the Testing environment (registered via TryAddSingleton in Program.cs). A Notify queue health check (NotifyQueueHealthCheck) is registered alongside so the host reports queue readiness.

5) Dependency matrix

What the Notifier host pulls from the Notify toolkit (verified from the project references):

Connectors are restart-time plug-ins loaded from plugins/notify (see ../notify/architecture.mdsection 1). The 2026-06-06 DORA boundary slice removed the Worker ProjectReference to the executable StellaOps.Notify.Connectors.DoraInfoSharing implementation. DoraInfoSharingRuntimeDispatcher now builds only the host-owned request, signer, verifier, and delivery ledger contracts from StellaOps.Notify.Connectors.DoraInfoSharing.Contracts; MountedDoraInfoSharingConnectorRuntime loads the implementation assembly from the configured plugins/notify directory through PluginHost and invokes DoraInfoSharingSubscriberDeliveryService, DoraInfoSharingExportService, and DoraInfoSharingDsseService through that contract surface.

The signed regulatory profile uses these runtime paths:

• Bundle root: /app/plugins/notify/regulatory/stellaops.notify.connector.dorainfosharing/
• Config root: /app/etc/plugins/notify/regulatory/stellaops.notify.connector.dorainfosharing/ for profile-local bundle configuration, with Worker runtime options bound from notifier:dora:infoSharing:connector.
• Trust root: /app/trust-roots/plugins/notify/cosign.pub, overridable through notifier:dora:infoSharing:connector:TrustRootPath.
• Probe report: /var/lib/stellaops/plugin-scratch/notifier/probe-report.json, overridable through StellaOps:Plugins:ScratchRoot.

When notifier:dora:infoSharing:connector:EnforceSignatureVerification=true, the mounted bundle must clear shared signed-runtime admission before executable code is loaded: the manifest.json id must equal stellaops.notify.connector.dorainfosharing, module must be notify, profile must be regulatory, contractVersion must be runtime-bundle.v1, capabilities must include notify:connector:dora-info-sharing, the assembly sha256 must match the manifest, and the adjacent <assembly>.sig must verify against the configured trust root. If the mounted assembly is absent, the Worker fails the DORA dispatch path before signing or subscriber transport with connector-bundle-not-mounted, and the probe report emits a blocking stellaops.notify.connector.dorainfosharing row when that plugin is requested. If the bundle is mounted but unsigned or signed by an untrusted key, the probe row is rejected with an actionable signature reason. devops/build/package-runtime-plugins.ps1 -Module notify -Profile regulatory emits the regulatory bundle metadata, and docker-compose.plugins.notify-connectors.yml mounts the regulatory profile into Notifier Worker. The remaining acceptance evidence is image audit proof that the implementation DLL exists only under /app/plugins plus live pluginized compose collector evidence.

The host also pulls non-Notify platform libraries: Router transport (StellaOps.Router.AspNet, StellaOps.Router.Transport.Messaging) and Authority integration (StellaOps.Auth.ServerIntegration) on the WebService; air-gap egress policy (StellaOps.AirGap.Policy), canonical JSON, cryptography, and worker-health on the Worker; audit emission and localization across both.

6) Deployment distinction

Deployed topology (corrected 2026-07-12). The 2025-11-02 module boundary decision intended src/Notifier/ to be the deployable Studio host; the delivered topology differs and this section records the delivered reality:

• src/Notifier/ ships exactly one deployed container: StellaOps.Notifier.Worker(notifier-worker, horizontally-scalable consumers). Its StellaOps.Notifier.WebService is commented out of compose (rollback-only, “MERGED into notify-web”) and does not ship.
• notify-web(src/Notify/StellaOps.Notify.WebService) is the deployed API host and serves the full Studio surface — v1 plus the complete v2 enterprise route set (Program.cs:453-468). The “toolkit vs host” framing does not match this: the src/Notify/ tree owns the real web deployable, and src/Notifier/ contributes the worker. See the API-versioning note in ../notify/architecture.md§0.

Keeping the two trees separate preserves packaging flexibility, Offline Kit parity (connectors, default templates, and seed rules ship as a plugins/notify tree the air-gap builder copies verbatim), and cross-module governance. The directories must not be merged without an approved packaging RFC.

Note on hostnames. In merged-console deployments the historical notifier.stella-ops.local hostname can appear as an alias on the notify-web container; the gateway routes Studio traffic through the notifier frontdoor onto the service-local API surface, which mixes /api/v2/{resource} resource groups (rules, templates, simulate, incidents, escalations, quiet-hours, throttles, overrides, security, storm-breaker, localization, ack, fallback) with the inline /api/v2/notify/{channels,deliveries} and /api/v1/* ingress routes. Routing/frontdoor details live in ../notify/architecture.md.

7) Cross-links

Toolkit (Notify):

• Toolkit architecture — ../notify/architecture.md(event model, rules engine, templates, schema, observability, air-gap bootstrap)
• Toolkit overview — ../notify/overview.md
• Rules — ../notify/rules.md
• Digests — ../notify/digests.md
• Templates — ../notify/templates.md
• Toolkit README — ../notify/README.md

Notifier host:

• Host README — README.md
• Dev SMTP loop (Mailpit fall-back) — operations/dev-smtp.md

Related modules:

• Authority (OAuth clients / OpTok enforcement) — ../authority/
• Scheduler (event sources) — ../scheduler/

8) Epic alignment

• Epic 11 – Notifications Studio: Notifier is the host that delivers the Studio experience — notifications workspace, preview/simulation tooling, immutable delivery ledger, throttling/digest controls, escalation/on-call, and correlation features — by composing the Notify toolkit.