StellaOps Notify

Audience: operators wiring up alerting, on-call leads tuning rules, and engineers integrating the notifications engine into their own services.

Notify — the Notifications Studio — turns platform events into tenant-scoped alerts with deterministic delivery, offline parity, and a full audit trail. It is the alerting surface of the Stella Ops release control plane: every scan result, policy budget breach, attestation, or compliance threshold can be routed to the right responders through chat, email, webhooks, paging tools, and regulatory connectors, without storms or noise.

The codebase spans two source trees:

• src/Notify/*— the reusable models, engine, persistence, queue, and connector libraries, plus the StellaOps.Notify.WebService HTTP host.
• src/Notifier/*— the worker runtime library (StellaOps.Notifier.Worker) and the parallel StellaOps.Notifier.WebService / worker hosts.

StellaOps.Notify.WebService references StellaOps.Notifier.Worker for its runtime services, so downstream systems can embed the rules engine without inheriting the Studio UI.

Scope & responsibilities

• Apply tenant-scoped rules to platform events. The canonical event kinds (NotifyEventKinds) cover Scanner (scanner.report.ready, scanner.scan.completed), Scheduler (scheduler.rescan.delta, scheduler.doctor.run_alert), Attestor (attestor.logged), Zastava (zastava.admission), Concelier/Excititor (conselier.export.completed, excitor.export.completed, concelier.federation.bundle.ready|failed), Policy budgets (policy.budget.exceeded|warning), AirGap (airgap.*), Findings (findings.production_alert.evidence_seeded), and NIS2 effectiveness (nis2.effectiveness.threshold_breached).
• Render channel-specific payloads (Slack, Teams, Email, Webhook, PagerDuty, OpsGenie, Discord, Telegram, in-app inbox, and regulatory connectors) using deterministic templates with localisation safeguards.
• Enforce throttling, digests, quiet-hour calendars, maintenance windows, storm-breaker suppression, and operator overrides so bursts stay explainable and recoverable.
• Persist deliveries, attempts, throttles, escalation/on-call state, and rendered-body hashes (NotifyDelivery.BodyHash) for CLI/UI investigation and compliance export.

Current capabilities

The runtime host is StellaOps.Notify.WebService (src/Notify/StellaOps.Notify.WebService). It hosts the HTTP API and references the shared worker library StellaOps.Notifier.Worker (src/Notifier/StellaOps.Notifier/StellaOps.Notifier.Worker) for correlation, dispatch, escalation, digesting, and retention services. (Note: there is no StellaOps.Notify.Worker project — the worker logic lives under the StellaOps.Notifier.* tree and is consumed by the Notify host via project reference.)

• Rules + templates + incidents API: StellaOps.Notify.WebService exposes CRUD, template preview/validate, incident list/ack/resolve, simulation, and an in-app inbox under /api/v2/notify/* and sibling /api/v2/* groups, secured by Authority scopes (notify.viewer / notify.operator / notify.admin / notify.escalate).
• Worker pipeline (StellaOps.Notifier.Worker): ingests bus events, evaluates NotifyRuleMatch predicates, correlates incidents (Correlation/CorrelationEngine, IncidentManager), applies per-tenant token-bucket throttles (NotifyThrottler), evaluates quiet-hours/maintenance windows, coalesces digests (Digest/DigestGenerator, DigestScheduleRunner), runs storm-breaker detection, and dispatches deliveries with fallback chains and dead-letter handling.
• Connector plug-ins: restart-time plug-ins under StellaOps.Notify.Connectors.* with health checks and metadata declared in notify-plugin.json. Implemented connectors: Slack, Teams, Email, Webhook, PagerDuty, OpsGenie, Discord, Telegram, InApp, InAppInbox, plus regulatory/compliance connectors (Csaf, Dora, DoraInfoSharing, Enisa, NCS) and a shared base (Connectors.Shared).
• Template engine: deterministic rendering (StellaOps.Notify.Engine) with safe helpers, locale bundles, redaction defaults, and a rendered-body SHA-256 hash (NotifyDelivery.BodyHash) for audit parity. Render modes: None, Markdown, Html, AdaptiveCard, PlainText, Json (NotifyTemplateRenderMode).
• Delivery ledger: PostgreSQL-backed (schema notify), with the notify.deliveries table partitioned monthly by created_at, storing rendered bodies/hashes, attempts, status, correlation links, external IDs, and retry timing. Row-level security enforces tenant isolation on every table via notify_app.require_current_tenant().

Implemented since the original Sprint 38/39 plan

The correlation/digest/simulation/quiet-hours work originally tracked as “Sprint 39 focus” (NOTIFY-SVC-39-001…004) has landed:

• Correlation engine, incident lifecycle, and token-bucket throttling — StellaOps.Notifier.Worker/Correlation/*.
• Digest generator with schedule runner — StellaOps.Notifier.Worker/Digest/*.
• Simulation / rule dry-run API — StellaOps.Notifier.Worker/Simulation/*, exposed at POST /api/v2/simulate.
• Quiet-hours calendar + maintenance windows with audit logging — StellaOps.Notifier.Worker/Correlation/QuietHours* and /api/v2/quiet-hours.

HTTP API surface

Protected routes require an Authority bearer token (audiences notify and stellaops) carrying a validated stellaops:tenant claim. A gateway-forwarded X-StellaOps-TenantId header may be present as signed envelope context, but Notify does not require or trust a raw tenant header as the data-isolation source. Endpoint groups are wired in Program.cs (MapNotifyApiV2, MapNotifyAdminCompatEndpoints, plus the per-feature Map*Endpoints calls):

A legacy envelope/internal compatibility surface (/api/v1/notify/*, /internal/notify/*) is also mapped in Program.cs for schema normalisation and the gateway envelope-bridge path. Status for remaining work is tracked in src/Notifier/StellaOps.Notifier/TASKS.md and sprint plans.

Key docs

• overview.md— summary of capabilities, imposed rules, and customer journey.
• architecture.md/ architecture-detail.md— Notifications Studio runtime view.
• rules.md— declarative matcher syntax and evaluation order.
• digests.md— digest windows, coalescing logic, and delivery samples.
• templates.md— template helpers, localisation, and redaction guidelines.
• channels.md/ escalations.md— connector configuration and escalation/on-call routing.

Integrations & dependencies

• Storage: PostgreSQL (schema notify, plus the notify_app helper schema) for channels, rules, templates, deliveries (monthly partitions), digests, quiet hours, maintenance windows, escalation policies/states, on-call schedules, inbox, incidents, audit, and locks. Schema is auto-migrated on startup via AddNotifyPersistence → AddStartupMigrations (migrations embedded in StellaOps.Notify.Persistence/Migrations/*.sql); seed migrations are gated behind the NOTIFY_BOOTSTRAP_ENABLED flag.
• Queues: the event/delivery queue (StellaOps.Notify.Queue) supports a Redis (default, NotifyQueueTransportKind.Redis) or NATS JetStream (NotifyQueueTransportKind.Nats) transport, configured under notify:queue. Subjects: events on notify.events with dead-letter notify.events.dead; deliveries on notify.deliveries with dead-letter notify.deliveries.dead.
• Authority: OpTok-protected APIs accepting audiences notify and stellaops. Scopes (canonical names in StellaOpsScopes): notify.viewer, notify.operator, notify.admin, and notify.escalate (escalation/on-call actions). Channel credentials are stored as secret references.
• Observability:
  • Worker meter StellaOps.Notifier (DefaultNotifyMetrics / NotifyMetricNames) emits counters notify.delivery.attempts, notify.escalation.events, notify.deadletter.entries, notify.rule.evaluations, notify.template.renders, notify.storm.events, notify.retention.cleanups; histograms notify.delivery.duration, notify.rule.evaluation.duration, notify.template.render.duration; and observable gauge notify.queue.depth.
  • Queue transport (NotifyQueueMetrics) emits notify_queue_enqueued_total, notify_queue_deduplicated_total, notify_queue_ack_total, notify_queue_retry_total, notify_queue_deadletter_total (tagged by transport).
  • ActivitySource StellaOps.Notifier for OTEL traces; dashboards documented in architecture-detail.md.

Operational notes

• Schema fixtures live in ./resources/schemas; event and delivery samples live in ./resources/samples for contract tests and UI mocks.
• Offline Kit bundles ship plug-ins, default templates, and seed rules; update manifests under ops/offline-kit/ when connectors change.
• Observability assets: operations/observability.mdand operations/dashboards/notify-observability.json (offline Grafana import). Coordinate before renaming metrics or labels — dashboards and alert references depend on the metric names listed above.
• When releasing new rule or connector features, update the guidance in this directory and the related checklists.

Epic alignment

• Epic 11 – Notifications Studio: notifications workspace, preview tooling, immutable delivery ledger, throttling/digest controls, and forthcoming correlation/simulation features.

Implementation Status

Delivery Phases

• Phase 1 – Core rules engine & delivery ledger: Implement rules/channels schema, event ingestion, rule evaluation, idempotent deliveries, audit logging
• Phase 2 – Connectors & rendering: Ship Slack/Teams/Email/Webhook connectors, template rendering, localization, throttling, retries, secret referencing
• Phase 3 – Console & CLI authoring: Provide UI/CLI for rule authoring, previews, channel health, delivery browsing, digests, test sends
• Phase 4 – Governance & observability: Add approvals, RBAC, tenant quotas, metrics/logs/traces, dashboards, alerts, runbooks
• Phase 5 – Offline & compliance: Produce Offline Kit bundles (rules/channels/deploy scripts), signed exports, retention policies, auditing

Acceptance Criteria

• Rules evaluate deterministically per event; deliveries idempotent with audit trail and a rendered-body SHA-256 hash (NotifyDelivery.BodyHash). (Note: deliveries are content-hashed, not DSSE-signed; DSSE/attestation is owned by the Attestor/Signer modules.)
• Channel connectors support retries, rate limits, health checks, previews; secrets referenced securely
• Console/CLI support rule creation, testing, digests, delivery browsing, export/import workflows
• Observability dashboards track delivery health; alerts fire for sustained failures or backlog; runbooks cover remediation
• Offline Kit bundle contains configs, rules, digests, deployment scripts for air-gapped installs
• Notify respects tenancy and RBAC; governance (approvals, change log) enforced for high-impact rules

Key Risks & Mitigations

• Notification storms: Throttling, digests, dedupe windows, preview/test gating
• Secret compromise: Secret references only, rotation workflows, audit logging
• Connector API changes: Versioned adapter layer, nightly health checks, fallback channels
• Noise vs signal: Simulation previews, metrics, rule scoring, recommended defaults
• Offline parity: Export/import of rules, connectors, digests with signed manifests

Current Phase Progress

The 2025-era phase notes below were stale; corrected against src/Notify/src/Notifier as of the latest reconciliation:

• Phase 1: Complete — rules/templates/channels schema, event ingestion, rule evaluation, partitioned delivery ledger, and audit logging are implemented.
• Phase 2: Complete — Slack/Teams/Email/Webhook connectors shipped alongside PagerDuty/OpsGenie/Discord/Telegram/in-app and regulatory connectors; template rendering, localization bundles, throttling, retries, and secret referencing are in place.
• Phase 3: Largely complete via the HTTP API — rule/template authoring, previews, channel listing, delivery browsing, digests, simulation, and test sends are exposed (/api/v2/*); standalone CLI/Console authoring coverage tracked separately.
• Phase 4: Observability (metrics/traces/dead-letters/retention/chaos) and RBAC scopes are implemented; approvals/quotas governance partially landed (pack-approval persistence, operator overrides).
• Phase 5: Offline Kit bundling and retention/auditing implemented; verify export/import signed-manifest details against the current Offline Kit pipeline before relying on them.