StellaOps Notify

Verified against commit 942a18727c on 2026-09-26 (UTC). Re-verify: grep -nE "^ notify-(web|worker):" devops/release/bundle/docker-compose.yml devops/compose/docker-compose.stella-services.yml devops/compose/docker-compose.notify.yml and ls src/Notify/__Libraries/StellaOps.Notify.Persistence.Consolidated/Migrations.

Service rows (host project, image, database and role, compose file, bundle presence, Doctor registration) are generated in docs/architecture/service-inventory.md; where this page and the inventory differ, the inventory is authoritative.

Audience: operators wiring up alerting, on-call leads tuning rules, and engineers integrating the notifications engine into their own services.

Notify turns platform events into tenant-scoped alerts with deterministic delivery, offline parity and an audit trail. Scan results, policy budget breaches, attestations, deployment and backup events, and compliance thresholds are routed to responders through chat, email, webhooks, paging tools, an in-app inbox and regulatory connectors, with throttling, digests and storm suppression.

The codebase is one source tree, src/Notify/:

• src/Notify/__Libraries/: models, contracts, engine, delivery runtime (StellaOps.Notify.Delivery), persistence, queue, and connector libraries.
• src/Notify/StellaOps.Notify.WebServiceand src/Notify/StellaOps.Notify.Worker: the two deployables, notify-web and notify-worker. Both depend on same-family libraries and never on each other.

Deployables

notify-worker is the only delivery consumer. Its delivery pipeline is default-off (Notify:DeliveryPipeline:Enabled, wired in src/Notify/StellaOps.Notify.Worker/NotifyDeliveryPipelineRegistration.cs); the activation overlay turns it on. The release bundle ships notify-web only, so a bundle install accepts, stores and serves notification configuration and the in-app inbox, and does not run the delivery worker.

Database

• One database, stellaops_notify, on the shared PostgreSQL server, owned by the notify role.
• Connection: STELLAOPS_POSTGRES_NOTIFY_CONNECTION only, fail-closed (src/Notify/__Libraries/StellaOps.Notify.Persistence.Consolidated/Extensions/ConsolidatedNotifyPersistenceExtensions.cs).
• Migration authority: StellaOps.Notify.Persistence.Consolidated/Migrations/ (001 to 010), applied at startup by both roles. The runtime DAL StellaOps.Notify.Persistence embeds no migration SQL; its four historical scripts are regression-test inputs only (RFC-9).
• Fresh-database admission (CIC-2). With Notify:SecretSealing:ExistingRowsConverged absent or false, NotifyCredentialConvergenceAdmissionGate admits a database whose notify.channels and notify.webhook_security_configs hold no row and records a receipt in notify.credential_convergence_admissions (migration 010); later starts admit on the receipt. A populated database without a receipt still refuses. The bundle sets NOTIFY_NOTIFY__SECRETSEALING__EXISTINGROWSCONVERGED to false by default.
• Migration 008 exception (CIC-2, owner ruling 2026-09-25). 008_notify_retention_mechanisms.sql was edited once under a recorded ADR-004 exception: its escalation backfill runs only when notify.escalation_states has a non-empty heap, so a fresh database converges under the NOBYPASSRLS runtime role. Estates that applied the original reconcile through NotifyMigrationChecksumPins. Register: ADR-004 exception register.

Doctor checks

notify-web serves the doctor-check/v1 endpoint behind the estate ops.health capability (MapServiceDoctorEndpoints in Program.cs). Notify-owned check ids: doctor.notify.rls-posture, doctor.notify.deliveries.partition-coverage, doctor.notify.retention.delivery-backlog, doctor.notify.retention.inbox-backlog, doctor.notify.retention.fallback-budget and doctor.notify.retention.mechanics-budget, alongside the SDK standard checks and the shared Valkey checks.

Scope & responsibilities

• Apply tenant-scoped rules to platform events. The event kind constants in NotifyEventKinds (src/Notify/__Libraries/StellaOps.Notify.Contracts/) cover Scanner (scanner.report.ready, scanner.scan.completed), Scheduler (scheduler.rescan.delta), Attestor (attestor.logged), policy budgets (policy.budget.exceeded|warning), AirGap (airgap.*), Findings (findings.production_alert.evidence_seeded), NIS2 effectiveness, Platform crypto-provider and backup events (platform.*), deployment and release events (deployment.*, release.*), and federation bundles (concelier.federation.bundle.ready|failed). Several constants keep predecessor producer names as wire literals (zastava.admission, conselier.export.completed, excitor.export.completed); they are event-name strings, not live services.
• Render channel-specific payloads (Slack, Teams, Email, Webhook, PagerDuty, OpsGenie, Discord, Telegram, in-app inbox, and regulatory connectors) using deterministic templates with localisation safeguards.
• Enforce throttling, digests, quiet-hour calendars, maintenance windows, storm-breaker suppression, and operator overrides so bursts stay explainable and recoverable.
• Persist deliveries, attempts, throttles, escalation/on-call state, and rendered-body hashes (NotifyDelivery.BodyHash) for CLI/UI investigation and compliance export.

Current capabilities

The API host is StellaOps.Notify.WebService (src/Notify/StellaOps.Notify.WebService) and the delivery host is StellaOps.Notify.Worker. Both reference the shared delivery library StellaOps.Notify.Delivery (src/Notify/__Libraries/StellaOps.Notify.Delivery) for correlation, dispatch, escalation, digesting, and retention services.

• Rules + templates + incidents API: StellaOps.Notify.WebService exposes CRUD, template preview/validate, incident list/ack/resolve, simulation, and an in-app inbox under /api/v2/notify/* and sibling /api/v2/* groups, secured by Authority scopes (notify.viewer / notify.operator / notify.admin / notify.escalate).
• Worker pipeline (StellaOps.Notify.Delivery, composed by notify-worker): ingests bus events, evaluates NotifyRuleMatch predicates, correlates incidents (Correlation/CorrelationEngine, IncidentManager), applies per-tenant token-bucket throttles (NotifyThrottler), evaluates quiet-hours/maintenance windows, coalesces digests (Digest/DigestGenerator, DigestScheduleRunner), runs storm-breaker detection, and dispatches deliveries with fallback chains and dead-letter handling.
• Connector plug-ins: restart-time plug-ins under StellaOps.Notify.Connectors.* with health checks and metadata declared in notify-plugin.json. Implemented connectors: Slack, Teams, Email, Webhook, PagerDuty, OpsGenie, Discord, Telegram, InApp, InAppInbox, plus regulatory/compliance connectors (Csaf, Dora, DoraInfoSharing, Enisa, NCS) and a shared base (Connectors.Shared).
• Template engine: deterministic rendering (StellaOps.Notify.Engine) with safe helpers, locale bundles, redaction defaults, and a rendered-body SHA-256 hash (NotifyDelivery.BodyHash) for audit parity. Render modes: None, Markdown, Html, AdaptiveCard, PlainText, Json (NotifyTemplateRenderMode).
• Delivery ledger: PostgreSQL-backed (schema notify), with the notify.deliveries table partitioned monthly by created_at, storing rendered bodies/hashes, attempts, status, correlation links, external IDs, and retry timing. Row-level security enforces tenant isolation on every table via notify_app.require_current_tenant().

Implemented since the original Sprint 38/39 plan

The correlation/digest/simulation/quiet-hours work originally tracked as “Sprint 39 focus” (NOTIFY-SVC-39-001…004) has landed:

• Correlation engine, incident lifecycle, and token-bucket throttling — StellaOps.Notify.Delivery/Correlation/*.
• Digest generator with schedule runner — StellaOps.Notify.Delivery/Digest/*.
• Simulation / rule dry-run API — StellaOps.Notify.Delivery/Simulation/*, exposed at POST /api/v2/simulate.
• Quiet-hours calendar + maintenance windows with audit logging — StellaOps.Notify.Delivery/Correlation/QuietHours* and /api/v2/quiet-hours.

HTTP API surface

Protected routes require an Authority bearer token (audiences notify and stellaops) carrying a validated stellaops:tenant claim. A gateway-forwarded X-StellaOps-TenantId header may be present as signed envelope context, but Notify does not require or trust a raw tenant header as the data-isolation source. Endpoint groups are wired in Program.cs (MapNotifyApiV2, MapNotifyAdminCompatEndpoints, plus the per-feature Map*Endpoints calls):

A legacy envelope/internal compatibility surface (/api/v1/notify/*, /internal/notify/*) is also mapped in Program.cs for schema normalisation and the gateway envelope-bridge path. Remaining work is tracked in docs/implplan/SPRINT_* rows.

Key docs

• overview.md— summary of capabilities, imposed rules, and customer journey.
• architecture.md: the architecture dossier.
• rules.md— declarative matcher syntax and evaluation order.
• digests.md— digest windows, coalescing logic, and delivery samples.
• templates.md— template helpers, localisation, and redaction guidelines.
• channels.md/ escalations.md— connector configuration and escalation/on-call routing.

Integrations & dependencies

• Storage: PostgreSQL (schema notify, plus the notify_app helper schema) for channels, rules, templates, deliveries (monthly partitions), digests, quiet hours, maintenance windows, escalation policies/states, on-call schedules, inbox, and incidents. Consolidated target hosts resolve STELLAOPS_POSTGRES_NOTIFY_CONNECTION fail-closed and auto-migrate from StellaOps.Notify.Persistence.Consolidated; they reuse the proven repositories in StellaOps.Notify.Persistence through the repository-only AddNotifyRuntimeRepositories registration. The runtime DAL embeds no migration SQL; four unchanged historical scripts are regression-test inputs only. See retention and cleanup for the current lifecycle boundaries. Clean target startup inserts no seed/demo rows.
• Queues: the event/delivery queue (StellaOps.Notify.Queue) supports a Redis (default, NotifyQueueTransportKind.Redis) or NATS JetStream (NotifyQueueTransportKind.Nats) transport, configured under notify:queue. Subjects: events on notify.events with dead-letter notify.events.dead; deliveries on notify.deliveries with dead-letter notify.deliveries.dead.
• Authority: OpTok-protected APIs accepting audiences notify and stellaops. Scopes (canonical names in StellaOpsScopes): notify.viewer, notify.operator, notify.admin, and notify.escalate (escalation/on-call actions). Channel credentials are stored as secret references.
• Observability:
  • Worker meter StellaOps.Notifier (DefaultNotifyMetrics / NotifyMetricNames) emits counters notify.delivery.attempts, notify.escalation.events, notify.deadletter.entries, notify.rule.evaluations, notify.template.renders, notify.storm.events, notify.retention.cleanups; histograms notify.delivery.duration, notify.rule.evaluation.duration, notify.template.render.duration; and observable gauge notify.queue.depth.
  • Queue transport (NotifyQueueMetrics) emits notify_queue_enqueued_total, notify_queue_deduplicated_total, notify_queue_ack_total, notify_queue_retry_total, notify_queue_deadletter_total (tagged by transport).
  • ActivitySource named by NotifierTracingOptions.SourceName (default StellaOps.Notifier) for OTEL traces; dashboards in operations/observability.md.

Operational notes

• Schema fixtures live in ./resources/schemas; event and delivery samples live in ./resources/samples for contract tests and UI mocks.
• Observability assets: operations/observability.mdand operations/dashboards/notify-observability.json (offline Grafana import). Coordinate before renaming metrics or labels — dashboards and alert references depend on the metric names listed above.
• When releasing new rule or connector features, update the guidance in this directory and the related checklists.

Retired predecessors

Subordinate documents