Notifications Hardening Guide

Last updated: 2026-05-30 (reconciled against src/Notify + src/Notifier implementation)

This guide documents the security controls protecting the Stella Ops notification path — channels, rules, templates, delivery history, and inbound/outbound webhooks across the Notify and Notifier services. It is written for operators hardening a deployment and for engineers reviewing notification-related changes. Each control cites the source symbol that implements it so claims can be verified against src/.

See also: secrets-handling.md, rate-limits.md, and authority-scopes.md.

Threat model

• Tenant data isolation breaches (cross-tenant deliveries).
• Channel compromise (webhook leaks, OAuth token theft).
• Message tampering or replay.
• Flooding / notification storms.

Controls

• Tenant isolation: every rule/channel/template/delivery row carries tenant_id; API handlers self-resolve the tenant from the X-StellaOps-TenantId header. PostgreSQL enforces tenant isolation at the database layer with PostgreSQL row-level security — every notify.* table has ENABLE ROW LEVEL SECURITY + FORCE ROW LEVEL SECURITY and a *_tenant_isolation policy of the form tenant_id = notify_app.require_current_tenant(), where the current tenant comes from the app.tenant_id session variable. A notify_admin role (NOLOGIN BYPASSRLS) exists for migration/maintenance only. Tenant-scoped B-tree indexes back the queries (e.g. idx_channels_tenant, idx_rules_enabled (tenant_id, enabled, priority DESC), ix_deliveries_part_status (tenant_id, status)), not a single (tenant_id, id) composite. (Source: StellaOps.Notify.Persistence/Migrations/001_initial_schema.sql.)
• Secrets: channels reference a credential indirectly via the channel config secretRef URI (for example vault://notify/channels/...); a credential resolver dereferences it before the connector runs, so plaintext secrets are not held in the canonical channel config. Rotate the underlying secret in your secret store / Authority. (Source: NotifyChannelConfig.SecretRef, EmailConnectorConfig.) NOTE: a legacy credentials JSONB column still exists on notify.channels for backward compatibility, and webhook HMAC secrets are stored per-channel in the webhook security config; treat those stores as sensitive.
• Outbound allowlist: restrict hosts/ports per tenant; defaults block public internet in air-gapped kits.
• Outbound webhook signing: outbound webhook deliveries are HMAC-signed over the request body. The signature header is configurable per channel (default X-Webhook-Signature; a prefix such as sha256= can be set for Slack-style receivers) and the signature format is hex (default), base64, or base64url. The HMAC algorithm is HMAC-SHA256 by default (SHA384/SHA512 selectable, and the regional crypto profile may substitute GOST/SM per CoC §7.4). Receivers verify the signature and, when a timestamp header is configured, reject requests older than MaxRequestAge (default 5 minutes / 300s). (Source: InMemoryWebhookSecurityService, WebhookSecurityConfig, WebhookSecurityOptions.)
• Inbound webhook validation: inbound external-platform webhooks (PagerDuty, OpsGenie) are gated by an HMAC signature filter that validates against the tenant-scoped secret; the HMAC primitive is supplied by the regional crypto plugin chain. (Source: StellaOps.Notify.WebHooks.Inbound, AddNotifyInboundWebhooks.)
• Replay protection: inbound/validated webhooks use timestamp max-age checks (default 5m) plus an in-memory nonce cache (default 10m, keyed on the signature) to reject duplicates. Inbound platform events de-duplicate on an Idempotency-Key header or a derived {producer}|{tenant}|{kind}|{eventId} key persisted as the delivery external_id. (Source: InMemoryWebhookSecurityService.ValidateAsync, GenericEventEndpointExtensions.) NOTE: there is no separate 24h delivery ledger de-duplicating on (channel, bodyHash), and escalation/ack tokens are HMAC-signed (not DSSE) — see “Token signing” below.
• Token signing: internal Notifier tokens (ack tokens, signing-service tokens) are HMAC-signed through the central crypto abstraction (HMAC-SHA256 on the world/fips/kcmvp/eidas profiles, GOST/SM substituted on those profiles). Ack tokens use an HKDF-derived key and carry a default 7-day expiry (30-day max); signing-service tokens default to 24h expiry with 30-day key rotation and a 90-day verification-overlap retention window so previously-signed tokens stay verifiable across a rotation. (Source: HmacAckTokenService, SigningService, LocalSigningKeyProvider.)
• Rate limits/throttles: ASP.NET token-bucket rate limiters protect the delivery-history list (notify-deliveries) and channel test-send (notify-test-send) endpoints. Tenant-scoped throttle configuration suppresses duplicate notifications with a default window of 900s (15 min) plus per-event-kind overrides and optional burst windows; quiet hours and maintenance windows suppress non-critical traffic. (Source: ConfigureRateLimiting in Program.cs, ThrottleEndpoints.)
• HTML sanitization: notification HTML is sanitized against named profiles to strip disallowed tags/attributes and prevent XSS; /api/v2/security/html/{sanitize,validate,strip} expose this. (Source: IHtmlSanitizer, DefaultHtmlSanitizer.)
• Templates sandboxed: the renderer is a closed, regex-based Handlebars-style engine supporting only {{property}}, {{#each}}, and {{#if}} substitution over the event JSON payload — there is no file/network access and no operator-supplied helper registration. HTML render mode auto-escapes payload-derived values. (Source: AdvancedTemplateRenderer.) NOTE: Markdown→HTML conversion is tracked as a follow-up XSS surface (SPRINT_20260430_013_Notify_template_renderer_escape.md).
• Logging/PII: payloads redacted based on rule labels; logs avoid full body, store hashes instead.
• Audit: mutating admin actions (escalation policies, on-call schedules/overrides, throttle config, start/stop/manual escalation, channel/rule changes) are wrapped with .Audited(...) and emitted to the Timeline service with module, action, resource, actor, and trace context. (Source: EscalationEndpoints, ThrottleEndpoints, AddAuditEmission.)

Deployment checklist

• [ ] Authority scopes notify.viewer, notify.operator, notify.admin, and notify.escalate configured; service accounts least-privilege. (The notify.operator policy also accepts notify.admin; the escalation endpoints accept notify.escalate, notify.operator, or notify.admin.)
• [ ] HTTPS everywhere; TLS 1.2+; HSTS on WebService front-door.
• [ ] Valkey protected by auth and network policy; PostgreSQL TLS + auth enabled.
• [ ] Outbound allowlists defined per environment; no wildcard *.
• [ ] Webhook receivers validate signatures and enforce host/IP allowlists.

Incident playbook (channel compromise)

1. Disable the affected channel. Re-POST the channel definition (POST {Api.BasePath}/channels, upsert semantics) with enabled=false, or delete it (DELETE {Api.BasePath}/channels/{channelId}). (There is no PATCH /channels/{id} route; channel writes are upserts.)
2. Rotate the secret in your secret store, or for webhook channels rotate the HMAC secret via POST /api/v2/security/keys/rotate (signing keys) / the webhook security service RotateSecret path; update the channel secretRef if it changed.
3. Search delivery history (GET {Api.BasePath}/deliveries, filterable by channelId/time range) for deliveries to the compromised endpoint and notify tenants if required.
4. Re-enable with new endpoint/secret after validation.

Offline/air-gap notes

• Ship channel manifests and secrets via sealed bundles; keep hash manifest with signed checksum.
• Disable any channel type not supported in the enclave (e.g., external Slack) and use in-app or file-drop channels instead.