Notify — Architecture (Notifications Toolkit)

Audience: Notify toolkit maintainers and integrators; operators wiring channels, rules, and digests. Companion: the deployable Notifications Studio host that composes this toolkit is documented in ../notifier/architecture.md.

Scope. Implementation‑ready architecture for Notify (aligned with Epic 11 – Notifications Studio): a rules‑driven, tenant‑aware notification service that consumes platform events (scan completed, report ready, rescan deltas, attestation logged, admission decisions, etc.), evaluates operator‑defined routing rules, renders channel‑specific messages (Slack/Teams/Email/Webhook/PagerDuty/OpsGenie), and delivers them reliably with idempotency, throttling, and digests. It is UI‑managed, auditable, and safe by default (no secrets leakage, no spam storms).

Recent changes & contract notes

The dated entries below form an append-only contract changelog — durability cutovers, route/compatibility decisions, and the focused tests that prove them. Skim for the latest deltas, then read the numbered sections (starting at §0) for the durable architecture.

0) Mission & boundaries

Mission. Convert facts from Stella Ops into actionable, noise-controlled signals where teams already live (chat, email, paging, and webhooks), with explainable reasons and deep links to the UI.

Boundaries.

1) Runtime shape & projects

The actual layout (verified against src/Notify/) is a single deployable WebService plus reusable libraries under __Libraries/. The delivery worker host lives in src/Notifier/(documented separately at docs/modules/notifier/architecture.md); there is no StellaOps.Notify.Worker project inside src/Notify/. Persistence is StellaOps.Notify.Persistence (not …Storage.Postgres).

src/Notify/
 ├─ StellaOps.Notify.WebService/        # REST host: core /api/v1/notify + merged /api/v2/notify (Studio)
 ├─ __Libraries/
 │   ├─ StellaOps.Notify.Engine/        # rules engine, templates, idempotency, digests, throttles, channel-connector contracts + dispatcher
 │   ├─ StellaOps.Notify.Models/        # DTOs (Rule, Channel, Event, Delivery, Template), schema-version migration
 │   ├─ StellaOps.Notify.Persistence/   # canonical PostgreSQL persistence (notify schema, embedded SQL migrations) + InMemory
 │   ├─ StellaOps.Notify.Storage.InMemory/ # in-memory repositories (Testing/offline)
 │   ├─ StellaOps.Notify.Queue/         # event-queue abstraction (Redis/Valkey Streams or NATS JetStream)
 │   ├─ StellaOps.Notify.WebHooks.Inbound/ # per-tenant inbound webhook signature gate (PagerDuty/OpsGenie acks)
 │   └─ StellaOps.Notify.Connectors.*/  # channel plug-ins (see §5)
 └─ __Tests/                            # unit/integration suites per library/connector

Deployables:

Dependencies: Authority (OpToks; DPoP/mTLS), PostgreSQL (notify schema), Redis/Valkey or NATS (event queue; NotifyQueueTransportKind = Redis | Nats), HTTP egress to Slack/Teams/Webhooks/PagerDuty/OpsGenie/Discord/Telegram, SMTP relay for Email.

Configuration. Notify.WebService bootstraps from the mounted runtime file /app/etc/notify/notify.yaml in shipped containers (copy one of the flat templates such as etc/notify.yaml.sample or etc/notify.prod.yaml into devops/etc/notify/notify.yaml before compose startup). Use storage.driver: postgres and provide postgres.notify options (connectionString, schemaName, pool sizing, timeouts). Authority settings follow the platform defaults—when running locally without Authority, set authority.enabled: false and supply developmentSigningKey so JWTs can be validated offline.

api.rateLimits exposes token-bucket controls for delivery history queries and test-send previews (deliveryHistory, testSend). Default values allow generous browsing while preventing accidental bursts; operators can relax/tighten the buckets per deployment.

Plug-ins and bundle classification. Notify channel connectors are packaged under <baseDirectory>/plugins/notify; module-local classification metadata lives at src/Notify/plugins/notify/notify-bundles.json. The base bundle is required and contains only StellaOps.Notify.Connectors.InApp and StellaOps.Notify.Connectors.InAppInbox, which cover in-product transient notifications plus the durable inbox/CLI read surface. External delivery connectors are optional overlays: email (Email), chat (Slack, Teams, Webhook, Discord, Telegram), and paging (PagerDuty, OpsGenie). Regulatory/offline connectors are an optional regulatory overlay (Csaf, Dora, DoraInfoSharing, Enisa, NCS). StellaOps.Notify.Connectors.Shared is dependency-only and is not a channel bundle by itself.

Notify discovery intentionally lists first-party connector entry assemblies and the custom lane StellaOps.Notify.Plugin.*.dll; it does not use a broad StellaOps.Notify.Connectors.*.dll glob because dependency-only assemblies such as StellaOps.Notify.Connectors.Shared.dll may be copied beside a connector for dependency resolution but must not be admitted or probed as plugins.

plugins:
  baseDirectory: "/app"
  directory: "/app/plugins/notify/base"
  ensureDirectoryExists: false
  requireReadOnlyDirectory: true
  enforceSignatureVerification: true
  signature:
    provider: "cosign"
    publicKeyPath: "/app/trust-roots/plugins/notify/cosign.pub"
    useRekorTransparencyLog: false
  requiredPlugins:
    - StellaOps.Notify.Connectors.InApp
    - StellaOps.Notify.Connectors.InAppInbox
  orderedPlugins:
    - StellaOps.Notify.Connectors.InApp
    - StellaOps.Notify.Connectors.InAppInbox
    - StellaOps.Notify.Connectors.Slack
    - StellaOps.Notify.Connectors.Teams
    - StellaOps.Notify.Connectors.Email
    - StellaOps.Notify.Connectors.Webhook

requiredPlugins fails Notify readiness when the base bundle is absent; /healthz remains liveness-only while /readyz returns 503 with the missing assembly names. Production compose also sets requireReadOnlyDirectory=true, so a writable mounted plugin root keeps readiness red and /internal/plugins/status reports status=rejected until the bundle is mounted :ro. If a mounted DLL is present but unsigned, signed by an untrusted key, or otherwise rejected before load, readiness fails with a plug-in admission reason and /internal/plugins/status reports status=rejected. If requiredPlugins or orderedPlugins omit the base pair, Notify post-configuration prepends them so Offline Kit load order is deterministic. Optional duplicates are resolved by the existing first-loaded-wins rule in the dispatcher/health maps; the configured order therefore decides the winner and later duplicates are logged and ignored.

The Offline Kit job copies the plugins/notify tree into the air-gapped bundle; the ordered list keeps connector manifests stable across environments. devops/build/package-runtime-plugins.ps1 -Module notify -Profile <base|email|chat|paging|regulatory|recommended|harness|bad-signature> produces the generated profile directories from the connector projects, and -SignNotifyBundles -NotifyCosignKeyPath <key> -NotifyCosignPublicKeyPath <pub> must be used before a bundle is runtime-admissible in production compose. Signing/trust metadata belongs beside each bundle manifest (notify-plugin.json, or generated manifest.json for DORA runtime adapters, plus checksums and <assembly>.dll.sig signature material) before a bundle is promoted from source build output into devops/plugins/notify/<profile>/<plugin-id>. The private key remains out of repo; the public key is copied to the mounted trust root as cosign.pub. Rekor transparency checks are disabled in the default compose profile so sealed/offline deployments verify against the mounted trust root rather than an external network.

In the hosted Notifier worker, delivery execution is split across deterministic dispatch paths: WebhookChannelDispatcher continues to handle chat/webhook routes, AdapterChannelDispatcher resolves Email, PagerDuty, and OpsGenie through IChannelAdapterFactory, and the DORA information-sharing runtime dispatcher handles only channels whose purpose is dora-info-sharing. The DORA dispatcher references StellaOps.Notify.Connectors.DoraInfoSharing.Contracts only; executable DORA export, DSSE, and subscriber-delivery services are loaded from the mounted Notify connector assembly through the Worker mounted-runtime adapter. When signed admission is enabled, the regulatory bundle must be rooted at /app/plugins/notify/regulatory/stellaops.notify.connector.dorainfosharing/, declare capability notify:connector:dora-info-sharing in manifest.json, match the assembly sha256, and verify the adjacent detached signature against /app/trust-roots/plugins/notify/cosign.pub or the configured trust root. The provider externalId emitted by adapter-backed channels must survive persistence so inbound webhook acknowledgements can be resolved after restart; the DORA dispatcher separately persists the connector-returned delivery ledger and subscriber receipt metadata.

Authority clients. Register two OAuth clients in StellaOps Authority: notify-web-dev (audience notify.dev) for development and notify-web (audience notify) for staging/production. Both require the notify.viewer, notify.operator, and notify.admin scopes (plus notify.escalate where escalation workflows are used) and use DPoP-bound client credentials (client_secret in the samples). The resource server also tolerates the shared stellaops audience alongside notify (see NotifyWebServiceOptions.AuthorityOptions.Audiences). Reference entries live in etc/authority.yaml.sample, with placeholder secrets under etc/secrets/notify-web*.secret.example.

2) Responsibilities

  1. Ingest platform events from internal bus with strong ordering per key (e.g., image digest).
  2. Evaluate rules (tenant‑scoped) with matchers: severity changes, namespaces, repos, labels, KEV flags, provider provenance (VEX), component keys, admission decisions, etc.
  3. Control noise: throttle, coalesce (digest windows), and dedupe via idempotency keys.
  4. Render channel‑specific messages using safe templates; include evidence and links.
  5. Deliver with retries/backoff; record outcome; expose delivery history to UI.
  6. Test paths (send test to channel targets) without touching live rules.
  7. Audit: log who configured what, when, and why a message was sent.

3) Event model (inputs)

Notify subscribes to the internal event bus (produced by services, escaped JSON; gzip allowed with caps):

Canonical envelope (bus → Notify.Engine):

{
  "eventId": "uuid",
  "kind": "scanner.report.ready",
  "tenant": "tenant-01",
  "ts": "2025-10-18T05:41:22Z",
  "actor": "scanner-webservice",
  "scope": { "namespace":"payments", "repo":"ghcr.io/acme/api", "digest":"sha256:..." },
  "payload": { /* kind-specific fields, see below */ }
}

Examples (payload cores):

4) Rules engine — semantics

Rule shape (simplified):

name: "high-critical-alerts-prod"
enabled: true
match:
  eventKinds: ["scanner.report.ready","scheduler.rescan.delta","zastava.admission"]
  namespaces: ["prod-*"]
  repos: ["ghcr.io/acme/*"]
  minSeverity: "high"            # min of new findings (delta context)
  kev: true                      # require KEV-tagged or allow any if false
  verdict: ["fail","deny"]       # filter for report/admission
  vex:
    includeRejectedJustifications: false    # notify only on accepted 'affected'
actions:
  - channel: "slack:sec-alerts"  # reference to Channel object
    template: "concise"
    throttle: "5m"
  - channel: "email:soc"
    digest: "hourly"
    template: "detailed"

Evaluation order

  1. Tenant check → discard if rule tenant ≠ event tenant.
  2. Kind filter → discard early.
  3. Scope match (namespace/repo/labels).
  4. Delta/severity gates (if event carries delta).
  5. VEX gate (drop if event’s finding is not affected under policy consensus unless rule says otherwise).
  6. Throttling/dedup (idempotency key) — skip if suppressed.
  7. Actions → enqueue per‑channel job(s).

Idempotency key: hash(ruleId | actionId | event.kind | scope.digest | delta.hash | day-bucket); ensures “same alert” doesn’t fire more than once within throttle window.

Digest windows: maintain per action a coalescer:

5) Channels & connectors (plug‑ins)

Channel config is two‑part: a Channel record (name, type, options) and a Secret reference (Vault/K8s Secret). Connectors are restart-time plug-ins discovered on service start (same manifest convention as Concelier/Excititor) and live under plugins/notify/<channel>/.

Channel connectors (verified against src/Notify/__Libraries/StellaOps.Notify.Connectors.*):

General-purpose channels:

The application-level NotifyChannelType enum (StellaOps.Notify.Models/NotifyEnums.cs) carries: Slack, Teams, Email, Webhook, Custom, PagerDuty, OpsGenie, Cli, InAppInbox, InApp, Discord, Telegram (string-serialised via JsonStringEnumConverter; Discord/Telegram appended last so existing serialized values are stable). By contrast, the PostgreSQL notify.channel_type enum intentionally carries only the original six external types (email, slack, teams, webhook, pagerduty, opsgenie). In-product channels (InApp/InAppInbox/Cli), the Custom type, and the Discord/Telegram connectors persist with channel_type = webhook while their true NotifyChannelType is preserved losslessly in the channel metadata JSON (round-tripped by ToChannelEntity/ToNotifyChannel).

Regulatory / compliance connectors (handoff adapters, not general broadcast channels — see §7.1 and the per-channel dossiers):

Connector contract. The consolidated plug-in seam is INotifyChannelConnector (in StellaOps.Notify.Engine/ChannelConnectorContracts.cs), which extends the two legacy provider interfaces so a single registration satisfies health, test-preview, and wire delivery:

public interface INotifyChannelConnector
    : INotifyChannelHealthProvider,   // ChannelHealthContracts.cs — connector self-check
      INotifyChannelTestProvider      // ChannelTestPreviewContracts.cs — test-send preview
{
    // Delivers a rendered notification; returns status + provider externalId + metadata.
    Task<ChannelDeliveryOutcome> DeliverAsync(ChannelDeliveryContext context, CancellationToken cancellationToken);
}

Adopting the consolidated contract is opt-in: existing Email/Slack/Teams/Webhook connectors keep their separate INotifyChannelHealthProvider / INotifyChannelTestProvider registrations. NotifyChannelDeliveryDispatcher (Engine) routes a rendered delivery to the registered connector for the channel type. (ChannelDeliveryStatus is Delivered | Failed | Skipped.)

For hosted external channels, Notifier worker adapters implement IChannelAdapter and are selected by AdapterChannelDispatcher. Those adapters must emit stable provider identifiers (externalId, incidentId where applicable) so the IAckBridge webhook path can recover correlation from persisted delivery rows instead of process-local memory.

DeliveryContext includes rendered content and raw event for audit.

Test-send previews. Plug-ins can optionally implement INotifyChannelTestProvider to shape /channels/{id}/test responses. Providers receive a sanitised ChannelTestPreviewContext (channel, tenant, target, timestamp, trace) and return a NotifyDeliveryRendered preview + metadata. When no provider is present, the host falls back to a generic preview so the endpoint always responds.

Secrets: ChannelConfig.secretRef is a unified secret reference (builtin:// / vault:// / openbao:// / authref:// / inline dev schemes) resolved at send-time by INotifyChannelSecretResolver through ISecretProvider (ADR-031/032) — see §11 Security. The live worker resolves it fail-closed immediately before the connector runs; plug-in manifests (notify-plugin.json) declare capabilities and version.

6) Templates & rendering

Template engine: strongly typed, safe Handlebars‑style; no arbitrary code. Partial templates per channel. Deterministic outputs (prop order, no locale drift unless requested).

Variables (examples):

Helpers:

Channel mapping:

i18n: template set per locale (English default; Bulgarian built‑in).

6.1) Placeholder substitution & auto-escape (H1a)

Placeholder substitution in AdvancedTemplateRenderer is format-aware. When template.RenderMode == Html, payload-derived values resolved by the substitution helpers are auto-escaped via System.Net.WebUtility.HtmlEncode before they replace the placeholder token. This applies to:

Markdown / PlainText / JSON modes are unchanged (no auto-escape) — they preserve the raw substituted string. Markdown bodies are author-controlled by template authors; downstream Markdown→HTML conversion (ConvertMarkdownToHtml) is a separate concern (see risk below). PlainText must not double-escape because it is delivered as text. JSON-mode bodies are wrapped in a JsonObject by ConvertToJson (Json+Json delivery is passthrough), and the surrounding JSON encoder already escapes appropriately.

Static template body is never encoded — only resolved placeholder values. Template authors emitting trusted HTML must place that markup in the static template body, not in payload fields. Triple-stash ({{{name}}}) opt-out for trusted HTML payloads is reserved as a follow-up enhancement; until it lands, all HTML-mode payload values are encoded.

Cross-link: the H1 finding in docs-archive/qa/audits/microservice-audit-pass2-2026-04-29.md is the source for this contract; implementation tracked in docs-archive/implplan/SPRINT_20260430_013_Notify_template_renderer_escape.md (NOTIFY-H1a-001 / -002 / -003).

Risks (carried as separate sprints):

7) Data model (PostgreSQL)

Canonical JSON Schemas for rules/channels/events live in docs/modules/notify/resources/schemas/. Sample payloads intended for tests/UI mock responses are captured in docs/modules/notify/resources/samples/.

Database: stellaops_notify (PostgreSQL), schema notify(plus the notify_app helper schema holding the require_current_tenant() function). The schema is created and migrated forward-only from embedded SQL resources in StellaOps.Notify.Persistence via AddNotifyPersistence → the Notify startup migration host; there is no manual init-script dependency. The live set is 001_v1_notify_baseline.sql (the collapsed pre-1.0 001–012 chain) + 002_sprint040_escalation_external_id_indexes.sql + the seed baseline S001_v1_notify_seed_baseline.sql; the old numbered files sit under Migrations/_archived/pre_1.0/mig061/ and are excluded from embedding (.csproj Exclude="Migrations\_archived\**\*.sql"), so cite the baseline, not the archived numbers. Every base table enforces row-level security keyed on the app.tenant_id session GUC (notify_admin is a BYPASSRLS maintenance role).

The shapes below are indicative; actual tables use concrete typed columns (PostgreSQL, not Mongo — the legacy _id shorthand maps to a uuid id primary key). rules, channels, and templates additionally carry a metadata JSONB column that losslessly round-trips the full canonical NotifyRule/NotifyChannel/NotifyTemplate model.

Indexes (verified against 001_v1_notify_baseline.sql): rules by {tenant_id} and {tenant_id, enabled, priority DESC}; deliveries by {tenant_id}, {tenant_id, status}, {status, next_retry_at} WHERE status IN ('pending','queued'), {channel_id}, {correlation_id} WHERE correlation_id IS NOT NULL, {tenant_id, created_at DESC} (most-recent-first listing — there is no sent_at index), a BRIN index on created_at, and {external_id} WHERE external_id IS NOT NULL; channels by {tenant_id} and {tenant_id, channel_type}; templates by {tenant_id}. The ack-bridge resolver path {channel_id, external_id} WHERE external_id IS NOT NULL (partial composite, partition-propagated) is created at baseline line 1225.

7.1 Reporting incident timeline state machine

IncidentReportTimelineService is the shared service contract for regulator incident reporting cadences. NIS2, CRA, and DORA are registered by default. NIS2 has three transmission milestones:

MilestoneState after sendDeadlineLedger event kindRequired evidence keys
early_warningEARLY_WARNING_SENTopened event + 24hnis2.incident.reportedincident_opened_event_ref, impact_summary_ref
notificationNOTIFICATION_SENTopened event + 72hnis2.incident.reportedincident_classified_event_ref, severity_classification_ref, operator_approval_ref
final_reportFINAL_REPORT_SENTopened event + 1 monthnis2.incident.reportedfinal_report_bundle_ref, early_warning_payload_hash, notification_payload_hash, operator_approval_ref

The same registry path registers DORA through CreateDoraRegime(initialReportOffset) with the default initial report offset set to 4 hours, followed by 72-hour intermediate and 1-month final report milestones:

MilestoneState after sendDeadlineLedger event kindRequired evidence keys
initial_reportEARLY_WARNING_SENTopened event + 4hdora.incident.reportedincident_opened_event_ref, dora_major_incident_classification_ref
intermediate_reportNOTIFICATION_SENTopened event + 72hdora.incident.reportedincident_classified_event_ref, intermediate_report_bundle_ref, operator_approval_ref
final_reportFINAL_REPORT_SENTopened event + 1 monthdora.incident.reportedfinal_report_bundle_ref, initial_report_payload_hash, intermediate_report_payload_hash, operator_approval_ref

CRA Article 14 reporting uses the same registry and starts its countdown from the cra.incident.classified ledger event so only CRA-reportable incidents enter the cadence:

MilestoneState after sendDeadlineLedger event kindRequired evidence keys
early_warningEARLY_WARNING_SENTclassified event + 24hcra.incident.reportedincident_classified_event_ref, exploitation_evidence_ref
vulnerability_notificationNOTIFICATION_SENTclassified event + 72hcra.incident.reportedincident_classified_event_ref, vulnerability_notification_bundle_ref, operator_approval_ref
final_reportFINAL_REPORT_SENTclassified event + 14dcra.incident.reportedfinal_report_bundle_ref, early_warning_payload_hash, vulnerability_notification_payload_hash, operator_approval_ref

Timeline inputs are expected to come from HLC-ordered ledger events. OpenedAt, ClassifiedAt, SentAt, and ClosedAt are supplied by the caller; TimeProvider is used only for evaluation of deadline status. Classification records the regime-specific classified ledger event kind/ref plus optional HLC value without advancing the reporting state. Each transmission records signer, normalized sha256:<hex> payload hash, source ledger event ref, optional HLC value, and sorted evidence refs. The service computes a deterministic ReplayHash from normalized timeline contents, including opened/classified/closed ledger refs, so frozen inputs produce stable replay evidence.

Late transmission after a deadline is rejected unless an operator override is supplied with actor, reason, timestamp, and optional ticket ref. Evaluation returns pending transmission schedules and a SEV-1 alert for the active milestone once 75 percent of the deadline has elapsed, or when the milestone is missed.

Timeline state is persisted in notify.incident_report_timeline_states through the Notify Postgres startup migration path. The row stores the normalized timeline JSON, current milestone deadline, deterministic ReplayHash, and ledger/audit refs extracted from the existing state-machine surface so a restarted host can reload the same shared NIS2 / CRA / DORA timeline without recalculating deadlines from process-local memory.

Nis2IncidentTimelineLedgerHandoffService is the Notify-owned handoff for the local Findings NIS2 ledger emitter. It opens, classifies, reports, and closes NIS2 incident timelines by calling INis2IncidentLedgerEventEmitter with caller-supplied event IDs/timestamps and then saving the Notify timeline only after the Findings append returns Success or Idempotent. Reported milestones carry the Notify notification timestamp, signer ref, authority/CSIRT intake ref, sha256:<hex> payload hash, evidence bundle ref, reachability subgraph refs, and VEX consensus refs into the nis2.incident.reported payload. The service does not synthesize ledger success: validation or conflict results fail the handoff before timeline persistence.

StellaOps.Notify.Connectors.Dora.DoraMajorIncidentOutboundChannel is the DORA major-incident handoff adapter. It encodes the pinned XBRL/iXBRL/envelope package, DSSE-signs the envelope bytes, writes the report artifacts plus a handoff manifest to an operator-controlled filesystem directory, and emits structured audit-event models for the caller to persist. Auto-submit remains default-off and is evaluated separately from the filesystem drop; production callers provide a normalized authorization snapshot derived from Authority’s GET /api/v1/tenants/{tenantId}/submission-authorizations?purpose=dora-incident&channelId=... projection. The connector does not call Authority directly. It accepts only the snapshot plus caller-supplied approval metadata, then fails closed unless auto-submit is enabled, an active Authority approver has submit or autoSubmitApproval, the approval scope matches tenant/channel/incident/report/variant, and the auth time is inside the default 5-minute fresh-auth window. The older channel-local notify.channel.dora.authorizedApprovers path remains available only as an offline deterministic unit-test fixture path.

StellaOps.Notify.Connectors.NCS.Nis2CsirtOutboundChannel is the NIS2 Article 23 CSIRT/NCA handoff adapter. It builds the local nis2-incident-report-envelope.v1 package, signs it through the SignerPipeline boundary, and produces a Notify delivery-ledger entry plus an audit-event model. The WebService operator route POST /api/v1/regulatory/nis2/incidents/{incidentId}/csirt-submissions invokes this adapter from a classified NIS2 timeline, persists the resulting Notify delivery row, and records the matching reporting milestone. DE, FR, and NL are represented as deterministic local portal stubs with operator-configured endpoints and blocked live-contract evidence. The generic signed-JSON webhook fallback is deliverable offline; PGP-email fallback prepares a deterministic manifest and fails closed until an approved local OpenPGP primitive is available through the Notify/crypto boundary.

8) External APIs (WebService)

StellaOps.Notify.WebService serves two route families on the same host (verified against src/Notify/StellaOps.Notify.WebService/Program.cs and Endpoints/NotifyApiEndpoints.cs):

Scopes (Authority OpToks). Authorization uses three core scopes plus an escalation scope — there is no notify.read/notify.write pair:

ScopePolicy constantGrants
notify.viewerNotifyViewer / NotifyPolicies.Viewerread-only: list/get channels, rules, templates, deliveries, digests, stats, observability
notify.operatorNotifyOperator / NotifyPolicies.Operatorwrite: create/update/delete rules, channels, templates, deliveries, digests, locks, audit entries, test-send, simulation, ack-token issue/verify
notify.adminNotifyAdmin / NotifyPolicies.Adminadministrative: security config, signing-key rotation, tenant-isolation grants, retention. Implicitly satisfies operator/viewer checks.
notify.escalateNotifyEscalateescalation actions: start/escalate/stop incidents, manage escalation policies and on-call schedules. Satisfied by operator/admin.

Scope names are configurable under notify:authority:{viewerScope,operatorScope,adminScope} (defaults above) and are wired by AddStellaOpsScopePolicy in Program.cs. Scope policy constants are defined in src/Notify/StellaOps.Notify.WebService/Constants/NotifierPolicies.cs (v2 surface) and Security/NotifyPolicies.cs (v1 surface).

REST calls that touch tenant data require the validated stellaops:tenant claim (matching the canonical tenantId stored in PostgreSQL). Gateway-forwarded tenant headers may still appear as envelope context, but Notify handlers do not use raw tenant headers as a data-isolation source. Payloads on the core surface are normalised via NotifySchemaMigration before persistence to guarantee schema version pinning.

Authentication is enforced through Authority resource-server OpTok validation when notify:authority:enabled=true (the production default). When Authority is disabled for offline/dev, a symmetric developmentSigningKey validates JWTs; notify:authority:allowAnonymousFallback=true (Development/Testing only) installs an allow-all handler. The tenant contract is identical in every mode: tenant scope comes from the validated stellaops:tenant claim, and any request body or query tenant must match it.

Service configuration exposes notify:authority:* keys (issuer, metadata address, audiences — default notify plus the shared stellaops audience, signing key, scope names) so operators can wire the Authority JWKS or (in dev) a symmetric test key. Postgres:Notify:* / notify:postgres:notify:* keys cover PostgreSQL connection/schema overrides, and notify:queue:* configures the event queue transport (see §13).

Runtime configuration validation. NotifyRuntimeConfigurationValidator (in src/Notify/StellaOps.Notify.WebService/Hosting/) implements the shared IRuntimeConfigurationValidatorcontract introduced in Sprint 20260513_018 URCV-004. It refuses to bring the Notify WebService up when notify:storage:driver=memory or notify:authority:allowAnonymousFallback=true are configured outside Development / Testing. The validator runs at host StartAsync via the shared RuntimeConfigurationValidationHostedService, replacing the prior inline static check that lived in Program.cs.

Internal tooling can hit /internal/notify/<entity>/normalize to upgrade legacy JSON and return canonical output used in the docs fixtures.

Note: the verbs above are listed relative to the core base path (/api/v1/notify). The merged /api/v2/notify family re-exposes rules/templates/incidents and adds the in-app inbox; admin/enterprise verbs (simulation, quiet-hours, throttles, overrides, escalation, on-call, storm-breaker, localization, fallback, security, observability, reporting-timeline, assurance-profile) are mapped under their own /api/v2/* prefixes (e.g. /api/v2/escalation-policies, /api/v2/oncall-schedules, /api/v2/security, /api/v2/ack), not under /api/v2/notify.

8.1 Ack tokens & escalation workflows

To support one-click acknowledgements from chat/email, the Notify WebService mints DSSE ack tokens via Authority. (Status note: the public route names below — /notify/ack-tokens/* — are the contract design; the merged WebService currently implements token sign/verify under /api/v2/security/* (with /keys/rotate for rotation) and ack processing/inbound webhooks under /api/v2/ack*. Treat the /notify/ack-tokens/* paths as the intended Authority-fronted surface rather than verified live routes.)

Authority signs ack tokens using keys configured under notifications.ackTokens. Public JWKS responses expose these keys with use: "notify-ack" and status: active|retired, enabling offline verification by the worker/UI/CLI.

Inbound PagerDuty and OpsGenie acknowledgement webhooks must resolve provider identifiers from durable delivery state (externalId plus incident metadata), not from process-local runtime maps. Restart-survival is a required property of the non-testing host composition.

8.1 Inbound external-platform webhooks (per-tenant signature gate)

External-platform acknowledgement webhooks (PagerDuty, OpsGenie) are gated by HMAC signature validation using a tenant-scoped secret resolved from the URL path segment. The legacy tenantless routes (/api/v2/ack/webhook/pagerduty, /api/v2/ack/webhook/opsgenie) have been removed (Sprint 20260430-005, audit finding A5); operators must reconfigure their PagerDuty / OpsGenie integrations to point at the per-tenant routes:

RouteMethodAuth
/api/v2/ack/webhook/pagerduty/{tenantId}POSTplatform HMAC signature (no bearer token)
/api/v2/ack/webhook/opsgenie/{tenantId}POSTplatform HMAC signature + X-OpsGenie-Webhook-Timestamp

Signature contract.

All HMAC operations route through StellaOps.Cryptography.IHmacAlgorithm, so regional crypto plugins (FIPS, GOST, SM, eIDAS) substitute the implementation transparently.

Tenant resolution. The validator reads the {tenantId} route segment only — never a header — and resolves the tenant against the configured Notify:Inbound:Tenants block. Multi-tenant deployments must configure one secret per tenant under Notify:Inbound:Tenants:<tenantId>:PagerDuty:Secret and Notify:Inbound:Tenants:<tenantId>:OpsGenie:Secret; values are SecretRef-style strings that the platform secret resolver expands (literal value, secret://, vault://).

Single-tenant fallback (audit-loud escape hatch). Legacy single-tenant deployments may enable Notify:Inbound:SingleTenantFallback:Enabled = true plus …:TenantId and …:PagerDuty:Secret / …:OpsGenie:Secret. Every use of the fallback emits a LogLevel.Critical audit event so accidental multi-tenant drift is loud. Multi-tenant deployments must leave this disabled.

Failure reasons (stable tokens). The filter returns 401 Unauthorized (or 404 Not Found for unknown tenant) with a problem-detail body whose reason field carries one of: tenant_unknown, tenant_missing, signature_missing, signature_invalid, timestamp_missing, timestamp_invalid, timestamp_too_old, timestamp_in_future, webhook_not_configured, platform_unsupported, ip_not_allowed. No secret material is ever reflected to the caller.

IP allowlist (defence-in-depth). Operators may configure a CIDR allow-list per platform via Notify:Inbound:PagerDutyAllowedSourceCidrs / Notify:Inbound:OpsGenieAllowedSourceCidrs. Empty list = allow-all. When non-empty, requests whose RemoteIpAddress is outside the list are rejected with reason = "ip_not_allowed".

Implementation lives in src/Notify/__Libraries/StellaOps.Notify.WebHooks.Inbound/ (filter + per-platform validators + secret resolver). The route registration is in src/Notify/StellaOps.Notify.WebService/Endpoints/EscalationEndpoints.cs.

Ingestion: workers do not expose public ingestion; they subscribe to the internal bus. The WebService producer bridge POST /api/v1/events/{producer} (in Extensions/GenericEventEndpointExtensions.cs, normally notify.operator scoped) is limited to allowlisted producer slugs and their required kind prefixes. The current allowlist includes concelier (concelier.*), attestor (attestor.*), findings (findings.*), nis2 (nis2.*), risk (risk.*), platform (platform.*), plus the explicitly any-kind scanner and release-orchestrator slugs. The literal /api/v1/events/platform route also accepts crypto:admin, crypto:profile:admin, or ops.admin because provider changes are composed inside those authenticated mutation requests; this exception does not authorize other producer slugs. The body tenant must equal the validated stellaops:tenant claim and eventId must be a non-empty GUID. Generic routes require the timestamp within ±5 minutes of the server clock (clock_skew). The literal Platform route preserves the durable outbox’s original occurrence time, accepts an old timestamp, and rejects a timestamp more than five minutes in the future; idempotent event admission remains mandatory. The findings.production_alert.evidence_seeded kind is handled specially (writes a queued durable delivery row); all other allowlisted kinds publish into the Notify event queue (notify:events stream, partitioned by tenant). Missing queue configuration is a terminal readiness failure (503 notify_event_queue_unavailable) rather than a best-effort/no-op acceptance. (Optional /events/test remains integration-test-only.)

9) Delivery pipeline (worker)

[Event bus] → [Ingestor] → [RuleMatcher] → [Throttle/Dedupe] → [DigestCoalescer] → [Renderer] → [Connector] → [Result]
                                                 └────────→ [DeliveryStore]

Idempotency: per action idempotency key stored in Valkey (TTL = throttle window or digest window). Connectors also respect provider idempotency where available (e.g., Slack client_msg_id).

10) Reliability & rate controls

11) Security & privacy

12) Observability (Prometheus + OTEL)

SLO targets

13) Configuration (YAML)

The authoritative WebService config keys are bound from the mounted runtime file etc/notify/notify.yaml inside the container content root (templates live as flat files such as etc/notify.yaml.sample plus notify.dev/stage/prod/airgap.yaml). Environment variables with the NOTIFY_ prefix override that file. The shape below reflects the actual NotifyWebServiceOptions (notify:storage, notify:postgres:notify, notify:authority, notify:api, notify:plugins, notify:telemetry) plus the event-queue options bound separately under notify:queue (NotifyEventQueueOptions; an optional notify:deliveryQueue mirrors it for the worker delivery stream). The limits/digests/quietHours/webhooks blocks below are illustrative pipeline tuning, not all part of NotifyWebServiceOptions:

storage:
  driver: postgres                # PostgreSQL-only after cutover; "memory" rejected outside Dev/Testing
postgres:
  notify:
    connectionString: "Host=postgres;Port=5432;Database=stellaops_notify;Username=stellaops;Password=stellaops;Pooling=true"
    schemaName: notify
    commandTimeoutSeconds: 30
authority:
  enabled: true
  issuer: "https://authority.stella-ops.local"
  metadataAddress: "https://authority.stella-ops.local/.well-known/openid-configuration"
  requireHttpsMetadata: true
  allowAnonymousFallback: false   # Dev/Testing only when true
  audiences: ["notify", "stellaops"]
  viewerScope: notify.viewer
  operatorScope: notify.operator
  adminScope: notify.admin
api:
  basePath: "/api/v1/notify"      # core toolkit surface; /api/v2/notify is mapped separately
  internalBasePath: "/internal/notify"
  tenantHeader: "X-StellaOps-TenantId" # gateway envelope context; handlers scope data by stellaops:tenant claim
notify:
  queue:
    transport: Redis              # NotifyQueueTransportKind: Redis (Valkey via redis:// protocol) | Nats
    redis:
      connectionString: "redis://valkey:6379"   # required when transport=Redis (else queue is "unavailable" → 503)
      streams:
        - { stream: "notify:events", consumerGroup: "notify-workers" }
    nats:
      url: "nats://nats:4222"     # required when transport=Nats
      stream: "NOTIFY_EVENTS"
      subject: "notify.events"
      deadLetterStream: "NOTIFY_EVENTS_DEAD"
  # --- illustrative pipeline tuning (not all in NotifyWebServiceOptions) ---
  limits:
    perTenantRpm: 600
    perChannel:
      slack:   { concurrency: 2 }
      teams:   { concurrency: 1 }
      email:   { concurrency: 8 }
      webhook: { concurrency: 8 }
  digests:
    defaultWindow: "1h"
    maxItems: 100
  quietHours:
    enabled: true
    window: "22:00-06:00"
    minSeverity: "critical"
  webhooks:
    sign:
      method: "ed25519"           # or "hmac-sha256"
      keyRef: "ref://notify/webhook-sign-key"

14) UI touch‑points

15) Failure modes & responses

ConditionBehavior
Slack 429 / Teams 429Respect Retry‑After, backoff with jitter, reduce concurrency
SMTP transient 4xxRetry up to maxAttempts; escalate to DLQ on exhaust
Invalid channel secretMark channel unhealthy; suppress sends; surface in UI
Rule explosion (matches everything)Safety valve: per‑tenant RPM caps; auto‑pause rule after X drops; UI alert
Bus outageBuffer to local queue (bounded); resume consuming when healthy
PostgreSQL slownessFall back to Valkey throttles; batch write deliveries; shed low‑priority notifications

16) Testing matrix

17) Sequences (representative)

A) New criticals after Concelier delta (Slack immediate + Email hourly digest)

sequenceDiagram
  autonumber
  participant SCH as Scheduler
  participant NO as Notify.Worker
  participant SL as Slack
  participant SMTP as Email

  SCH->>NO: bus event scheduler.rescan.delta { newCritical:1, digest:sha256:... }
  NO->>NO: match rules (Slack immediate; Email hourly digest)
  NO->>SL: chat.postMessage (concise)
  SL-->>NO: 200 OK
  NO->>NO: append to digest window (email:soc)
  Note over NO: At window close → render digest email
  NO->>SMTP: send email (detailed digest)
  SMTP-->>NO: 250 OK

B) Admission deny (Teams card + Webhook)

sequenceDiagram
  autonumber
  participant ZA as Zastava
  participant NO as Notify.Worker
  participant TE as Teams
  participant WH as Webhook

  ZA->>NO: bus event zastava.admission { decision: "deny", reasons: [...] }
  NO->>TE: POST adaptive card
  TE-->>NO: 200 OK
  NO->>WH: POST JSON (signed)
  WH-->>NO: 2xx

18) Implementation notes

19) Air-gapped bootstrap configuration

Air-gapped deployments ship a deterministic Notifier profile inside the Bootstrap Pack. The artefacts live under bootstrap/notify/ after running the Offline Kit builder and include:

These files are copied automatically by ops/offline-kit/build_offline_kit.py via copy_bootstrap_configs. Operators mount the configuration and secret into the StellaOps.Notify.WebService container (Compose or Offline Kit) to keep sealed-mode roll-outs reproducible. (Notifier WebService was merged into Notify WebService; the notifier.stella-ops.local hostname is now an alias on the notify-web container.)

20) Roadmap (post-v1)