Notification Flow

Overview

The Notification Flow describes how StellaOps delivers alerts and notifications to users through multiple channels including email, Slack, Microsoft Teams, webhooks, PagerDuty, OpsGenie, Discord, Telegram, and an in-app inbox. Notifications are triggered by platform events such as scan/report completion, rescan deltas, attestor log entries, admission decisions, export completions, air-gap drift/import, policy budget thresholds, and federation bundle readiness.

Audience: platform operators and integrators wiring notification channels, and engineers building event producers that feed the Notify subsystem.

Business Value: Timely notifications ensure security teams are immediately aware of critical issues without constantly monitoring dashboards.

Service topology note. The HTTP control surface (rules, templates, channels, incidents, escalations, the in-app inbox, and event ingestion) lives in the StellaOps.Notify.WebService project under src/Notify/. The background delivery worker — rule evaluation, template rendering, channel dispatch, retries, throttling, digests, escalations, and the delivery ledger — lives in StellaOps.Notifier.Worker under src/Notifier/. The two are deployed together as the Notify/Notifier subsystem; this document treats them as one logical “Notify” service.

Actors

Prerequisites

• An event queue transport is configured. Event ingestion (POST /api/v1/events/{producer}) returns 503 with error: notify_event_queue_unavailable when no queue is wired. Supported transports are Redis and NATS (StellaOps.Notify.Queue).
• Notification channels configured per tenant (rules, templates, channels CRUD)
• Channel credentials resolved via RFC-0001 secret references (config.secretRef) — SMTP password, Slack/Teams webhook tokens, PagerDuty routing key, etc. are never stored as plaintext config
• Rules defined to match event kinds and route to channel actions

Supported Channels

The connector set is the NotifyChannelType enum in StellaOps.Notify.Models/NotifyEnums.cs. Channel-specific connection details live in channel.config.properties (a string map); secrets are dereferenced from channel.config.secretRef by the RFC-0001 credential resolver before a connector runs.

NOT a separate channel type in code: there is no pagerduty.routing_key-style top-level config namespace, and smtp.host/slack.webhook_url/teams.webhook_url keys (used in earlier drafts of this doc) do not exist — the Email connector reads smtpHost/smtpPort/etc. directly. Regulatory delivery (CSAF, ENISA, DORA, NCS, NIS2/CSIRT) is expressed via NotifyChannelPurpose (dora-incident, dora-roi-distribution, dora-info-sharing, nis2-csirt, enisa-cra) on a channel rather than as distinct NotifyChannelType values; those purposes ride on top of webhook/email connectors.

Flow Diagram

The diagram below is a simplified illustration. In the implemented architecture the “Scheduler” lane is just one of many event producers (Scanner, Attestor, Zastava, Concelier/Excititor, Policy, Findings, AirGap), events flow through the event queue rather than a direct call, and the Notify lane is split between Notify.WebService (ingest/API) and Notifier.Worker (evaluate/render/dispatch).

┌─────────────────────────────────────────────────────────────────────────────────┐
│                            Notification Flow                                     │
└─────────────────────────────────────────────────────────────────────────────────┘

┌──────────┐  ┌───────────┐  ┌────────┐  ┌─────────────────────────────────────────┐
│  Event   │  │ Scheduler │  │ Notify │  │            Channel Adapters             │
│  Source  │  │           │  │        │  │  Email │ Slack │ Teams │ Webhook │ PD   │
└────┬─────┘  └─────┬─────┘  └───┬────┘  └────┬───┴───┬───┴───┬───┴────┬────┴──┬──┘
     │              │            │            │       │       │        │       │
     │ Emit event   │            │            │       │       │        │       │
     │─────────────────────────>│            │       │       │        │       │
     │              │            │            │       │       │        │       │
     │              │            │ Match      │       │       │        │       │
     │              │            │ rules      │       │       │        │       │
     │              │            │───┐        │       │       │        │       │
     │              │            │   │        │       │       │        │       │
     │              │            │<──┘        │       │       │        │       │
     │              │            │            │       │       │        │       │
     │              │            │ Route to   │       │       │        │       │
     │              │            │ channels   │       │       │        │       │
     │              │            │───────────>│       │       │        │       │
     │              │            │            │       │       │        │       │
     │              │            │            │ Send  │       │        │       │
     │              │            │            │ email │       │        │       │
     │              │            │            │──┐    │       │        │       │
     │              │            │            │  │    │       │        │       │
     │              │            │            │<─┘    │       │        │       │
     │              │            │            │       │       │        │       │
     │              │            │───────────────────>│       │        │       │
     │              │            │            │       │       │        │       │
     │              │            │            │       │ Post  │        │       │
     │              │            │            │       │ msg   │        │       │
     │              │            │            │       │──┐    │        │       │
     │              │            │            │       │  │    │        │       │
     │              │            │            │       │<─┘    │        │       │
     │              │            │            │       │       │        │       │
     │              │            │───────────────────────────>│        │       │
     │              │            │            │       │       │        │       │
     │              │            │            │       │       │ Post   │       │
     │              │            │            │       │       │ card   │       │
     │              │            │            │       │       │──┐     │       │
     │              │            │            │       │       │  │     │       │
     │              │            │            │       │       │<─┘     │       │
     │              │            │            │       │       │        │       │
     │              │            │ Log        │       │       │        │       │
     │              │            │ delivery   │       │       │        │       │
     │              │            │───┐        │       │       │        │       │
     │              │            │   │        │       │       │        │       │
     │              │            │<──┘        │       │       │        │       │
     │              │            │            │       │       │        │       │

Step-by-Step

1. Event Emission and Ingestion

Producer services emit canonical platform events. The wire shape is the NotifyEvent envelope (StellaOps.Notify.Models/NotifyEvent.cs): eventId (GUID), kind (lower-cased event kind), tenant, ts (UTC ISO-8601), payload (free-form JSON), and optional scope, version, actor, and attributes.

Events reach Notify by being published onto the event queue. One in-band path is the generic event endpoint POST /api/v1/events/{producer} (GenericEventEndpointExtensions), which is normally scoped to notify.operator, requires an envelope-bound tenant claim that matches the body tenant, enforces an allowlisted producer/kind relationship (concelier., attestor., findings., nis2., risk., platform., plus explicitly any-kind scanner/release-orchestrator lanes), rejects clock skew beyond ±5 minutes, and accepts an optional Idempotency-Key. The literal /api/v1/events/platform route additionally admits crypto:admin, crypto:profile:admin, or ops.admin for the authenticated provider-mutation composition path; those scopes do not open other producers. On success it enqueues to stream notify:events and returns 202 Accepted.

{
  "eventId": "0f3a9b2c-1234-5678-9abc-def012345678",
  "kind": "scanner.report.ready",
  "tenant": "acme-corp",
  "ts": "2024-12-29T10:30:00Z",
  "version": "1",
  "actor": "scanner:report-worker",
  "scope": {
    "repo": "library/nginx",
    "digest": "sha256:...",
    "image": "docker.io/library/nginx:1.25"
  },
  "payload": {
    "verdict": "fail",
    "severity": "critical"
  },
  "attributes": {
    "producer": "scanner",
    "correlationId": "..."
  }
}

The legacy fields event_type, event_id, timestamp, tenant_id, and metadata used in earlier drafts of this doc do not exist in the model — the canonical names are kind, eventId, ts, tenant, and attributes.

2. Notification Rule Matching

The Notifier worker evaluates each event against the tenant’s enabled rules (NotifyRule / NotifyRuleMatch in StellaOps.Notify.Models/NotifyRule.cs). A rule’s match block is a set of inclusion filters (all populated filters must be satisfied), not a generic field/operator/value DSL:

Each rule carries one or more actions (NotifyRuleAction): a target channel, optional template, optional digest window, optional throttle duration, optional locale, and an enabled flag.

// NotifyRule (conceptual JSON shape)
{
  "ruleId": "critical-scan-failure",
  "tenantId": "acme-corp",
  "name": "Critical scan failure",
  "enabled": true,
  "match": {
    "eventKinds": ["scanner.report.ready"],
    "verdicts": ["fail"],
    "minSeverity": "critical",
    "kevOnly": false
  },
  "actions": [
    { "actionId": "slack-secops", "channel": "chn-security-slack", "template": "tpl-critical", "throttle": "00:05:00" },
    { "actionId": "pd-oncall",    "channel": "chn-pagerduty-oncall" }
  ]
}

3. Channel Routing

For each matched rule, the worker fans out to the channels named by the rule’s enabled actions, applying any per-action throttle/digest window. Channels are resolved by channel id against the tenant’s configured NotifyChannel records; disabled channels are skipped. Priority/urgency is a connector-level concern (e.g. PagerDuty severity), not a first-class rule field.

4. Message Rendering

Each channel connector renders the message in the appropriate format. The rendered output is captured as NotifyDeliveryRendered on the delivery ledger (channelType, format, target, title, body, optional summary/textBody/locale/bodyHash/attachments). Templates declare a NotifyTemplateRenderMode — None, Markdown, Html, AdaptiveCard, PlainText, or Json — and the resulting NotifyDeliveryFormat mirrors the channel (Slack, Teams, Email, Webhook, PagerDuty, OpsGenie, Discord, Telegram, InApp, InAppInbox, Cli, plus Markdown/Html/PlainText/Json).

The examples below are illustrative of the per-channel output shape (not committed fixtures):

Email Template

<!DOCTYPE html>
<html>
<body>
  <h1>🚨 Critical Vulnerability Detected</h1>
  <p>A scan has failed with critical vulnerabilities.</p>

  <table>
    <tr><th>Image</th><td>docker.io/library/nginx:1.25</td></tr>
    <tr><th>Verdict</th><td style="color:red">FAIL</td></tr>
    <tr><th>Critical</th><td>1</td></tr>
    <tr><th>High</th><td>2</td></tr>
  </table>

  <p><a href="https://console.stellaops.local/scans/scan-7f3a9b2c-...">View Details</a></p>
</body>
</html>

Slack Block Kit

{
  "blocks": [
    {
      "type": "header",
      "text": {
        "type": "plain_text",
        "text": "🚨 Critical Vulnerability Detected"
      }
    },
    {
      "type": "section",
      "fields": [
        {"type": "mrkdwn", "text": "*Image:*\n`nginx:1.25`"},
        {"type": "mrkdwn", "text": "*Verdict:*\n:x: FAIL"},
        {"type": "mrkdwn", "text": "*Critical:*\n1"},
        {"type": "mrkdwn", "text": "*High:*\n2"}
      ]
    },
    {
      "type": "actions",
      "elements": [
        {
          "type": "button",
          "text": {"type": "plain_text", "text": "View Details"},
          "url": "https://console.stellaops.local/scans/scan-7f3a9b2c-..."
        }
      ]
    }
  ]
}

Teams Adaptive Card

{
  "type": "AdaptiveCard",
  "version": "1.4",
  "body": [
    {
      "type": "TextBlock",
      "text": "🚨 Critical Vulnerability Detected",
      "weight": "bolder",
      "size": "large"
    },
    {
      "type": "FactSet",
      "facts": [
        {"title": "Image", "value": "nginx:1.25"},
        {"title": "Verdict", "value": "FAIL"},
        {"title": "Critical", "value": "1"},
        {"title": "High", "value": "2"}
      ]
    }
  ],
  "actions": [
    {
      "type": "Action.OpenUrl",
      "title": "View Details",
      "url": "https://console.stellaops.local/scans/scan-7f3a9b2c-..."
    }
  ]
}

5. Delivery

Each connector delivers to its channel through the shared delivery transport (StellaOps.Notify.Connectors.Shared/DeliveryTransportOptions.cs). Retry behaviour is unified across HTTP-based connectors, not per-channel: the same exponential-backoff policy applies, tunable per channel via config.properties.

Backoff between attempt n and the next is deterministic: baseBackoff * 2^(n-1), capped at 5 minutes, no jitter (connector tests assert exact delay counts). Email uses System.Net SMTP rather than HTTP POST but shares the same attempt/backoff semantics.

The fixed per-channel retry tables in earlier drafts (e.g. “Slack 3 retries 1s/2s/4s”, “PagerDuty 5 retries 2s/4s/8s/16s/32s”, “Webhook 5 retries”) were illustrative and do not match the code — all connectors share the DeliveryTransportOptions policy above.

6. Delivery Logging

The worker records every delivery in the notify.deliveries ledger (a partitioned PostgreSQL table; see StellaOps.Notify.Persistence/Migrations/001_initial_schema.sql). The domain model is NotifyDelivery (StellaOps.Notify.Models/NotifyDelivery.cs): deliveryId, tenantId, ruleId, actionId, eventId, kind, status, statusReason, the rendered payload snapshot (rendered), the per-attempts array, and first-class externalId / channelId columns used by the ack-bridge resolver.

status is the NotifyDeliveryStatus enum: Pending, Queued, Sending, Delivered, Sent, Failed, Throttled, Digested, Dropped. Each attempt carries a NotifyDeliveryAttemptStatus: Enqueued, Sending, Succeeded, Failed, Throttled, Skipped.

// NotifyDelivery (conceptual JSON shape)
{
  "deliveryId": "dlv-789abc",
  "tenantId": "acme-corp",
  "ruleId": "critical-scan-failure",
  "actionId": "slack-secops",
  "eventId": "0f3a9b2c-1234-5678-9abc-def012345678",
  "kind": "scanner.report.ready",
  "status": "Delivered",
  "channelId": "chn-security-slack",
  "externalId": null,
  "attempts": [
    { "timestamp": "2024-12-29T10:30:02Z", "status": "Succeeded", "statusCode": 200 }
  ],
  "sentAt": "2024-12-29T10:30:02Z",
  "completedAt": "2024-12-29T10:30:02Z"
}

The table is notify.deliveries, not notify.delivery_log.

Data Contracts

Notification Event Schema (NotifyEvent)

interface NotifyEvent {
  eventId: string;            // GUID
  kind: string;               // lower-cased event kind (see Event Kinds)
  tenant: string;
  ts: string;                 // UTC ISO-8601
  payload?: unknown;          // free-form JSON node
  scope?: NotifyEventScope;   // namespace/repo/digest/component/image + labels/attributes
  version?: string;
  actor?: string;
  attributes?: Record<string, string>;
}

interface NotifyEventScope {
  namespace?: string;
  repo?: string;
  digest?: string;
  component?: string;
  image?: string;
  labels?: Record<string, string>;
  attributes?: Record<string, string>;
}

Channel Configuration Schema (NotifyChannel)

The channel record carries a typed type, a free-form config block, and a declared purpose. Connection details live in config.properties (a string→string map); the destination is config.target/config.endpoint; secrets are referenced by config.secretRef and resolved at delivery time — never inlined.

interface NotifyChannel {
  channelId: string;
  tenantId: string;
  name: string;
  type: 'Slack' | 'Teams' | 'Email' | 'Webhook' | 'Custom'
      | 'PagerDuty' | 'OpsGenie' | 'Cli' | 'InAppInbox' | 'InApp'
      | 'Discord' | 'Telegram';
  purpose: 'general' | 'dora-incident' | 'dora-roi-distribution'
         | 'dora-info-sharing' | 'nis2-csirt' | 'enisa-cra';
  enabled: boolean;
  config: NotifyChannelConfig;
  // displayName?, description?, labels?, metadata?, audit fields ...
}

interface NotifyChannelConfig {
  secretRef: string;                       // RFC-0001 credential reference (required)
  target?: string;                         // destination (e.g. recipients, Telegram chat id)
  endpoint?: string;                       // e.g. webhook URL
  properties?: Record<string, string>;     // connector-specific keys
  limits?: NotifyChannelLimits;            // concurrency / requestsPerMinute / timeout / maxBatchSize
}

Connector-specific keys live under properties. For example, the Email connector reads smtpHost, smtpPort, fromAddress, fromName, username, enableSsl (password resolved from secretRef); the shared delivery transport reads delivery.maxAttempts, delivery.baseBackoffMs, delivery.timeoutSeconds.

Event Kinds

Notify recognises the kinds declared in NotifyEventKinds (StellaOps.Notify.Models/NotifyEventKinds.cs). The kind string is the wire identifier matched by a rule’s eventKinds.

The event types in earlier drafts (scan.complete, policy.violation, advisory.new, advisory.update, vex.issued, exception.expiring, scheduled.daily_summary, scheduled.weekly_report) are not implemented — none appear in NotifyEventKinds. They are retained here only as a note of historical divergence. Note also the legacy misspellings preserved in the constants: conselier.export.completed and excitor.export.completed.

Error Handling

A failed send is retried up to delivery.maxAttempts (default 3) with the shared exponential backoff (default 200 ms base, ×2 per attempt, capped at 5 min) and a per-attempt timeout (default 30 s). Exhausted deliveries transition to Failed and may be dead-lettered (notify.dead_letter_entries, surfaced via notifier.deadletter.count). Storm/fallback handling and tenant suppression apply on top.

The per-channel recovery rows in earlier drafts (e.g. “Slack 429 → respect Retry-After”, “Teams 502 → retry up to 3 times”) describe desirable connector behaviour but are not differentiated in the shared transport; verify any connector-specific status handling against the relevant *ChannelConnector.cs before relying on it.

Authorization

Notify HTTP endpoints are gated by NotifierPolicies (StellaOps.Notify.WebService/Constants/NotifierPolicies.cs), which map directly to scopes in the canonical catalog (StellaOpsScopes.cs):

Most route groups (e.g. /api/v2/notify, /api/v2/rules, /api/v2/escalation-policies) require notify.viewer on the group and step up to notify.operator/notify.escalate/notify.admin on mutating routes.

HTTP surface (selected route prefixes)

• /api/v2/notify/rules, /templates, /incidents, /inbox (NotifyApiEndpoints)
• /api/v2/rules (RuleEndpoints), /api/v2/quiet-hours, /api/v2/overrides, /api/v2/simulate, /api/v2/fallback, /api/v2/security, /api/v2/localization
• /api/v2/escalation-policies, /api/v2/oncall-schedules, /api/v2/escalations, /api/v2/ack, /api/v2/ack/webhook (EscalationEndpoints)
• /api/v2/notify/channels, /deliveries, /quiet-hours, /throttle-configs, /escalation-policies, /overrides (NotifyAdminCompatEndpoints)
• /api/v1/events/{producer} (event ingestion), /api/v1/observability, /api/v1/notify/anomaly-subscriptions, /api/v1/regulatory/reporting-timelines, /api/v1/assurance/reporting-timeline-profiles

Observability

Metrics

Worker delivery metrics are emitted from the StellaOps.Notifier meter (StellaOps.Notifier.Worker/Observability); queue metrics from StellaOps.Notify.Queue.

The metric names in earlier drafts (notify_events_received_total, notify_deliveries_total, notify_delivery_latency_ms, notify_retries_total) are illustrative and do not match the instrument names registered in code.

Key Log Events

Structured log event names below are illustrative of the worker’s flow stages; treat the metric and route tables above as the load-bearing contract.

Related Flows

• Scan Submission Flow - Triggers scan notifications
• Advisory Drift Re-scan Flow - Advisory notifications
• Exception Approval Workflow - Exception notifications