E2E Notify-Channel-Coverage Harness — Operator Runbook

Audience: operators and CI maintainers running the Notify channel end-to-end coverage gate, plus engineers diagnosing a failed run.

StellaOps.E2E.NotifyChannelCoverage is the Notify-channel-coverage end-to-end harness — a sibling to StellaOps.E2E.AnalyzerCoverage for the Notify channel plugin family: Discord, Slack, Teams, email, webhook, opsgenie, pagerduty, telegram.

Notify channels deliver notifications; they do not analyze repos, so this harness has no git corpus. Instead it drives each channel connector against a mock receiver and asserts delivery + payload shape.

1. What it does — the four phases

PhaseNameNeeds receivers?What it asserts
1Connector test runnerNoEach Notify connector test project (Email, Slack, Teams, Webhook, Discord, Telegram, OpsGenie, PagerDuty, InApp, InAppInbox) passes via dotnet exec. Connector-absent channels (cli, a read-surface over InAppInbox) are skipped.
2Channel deliveryYesA deterministic synthetic alert is rendered into each channel’s wire envelope, routed through the real connector DeliverAsync seam, and delivered to a mock receiver — email → Mailpit (SMTP), webhook-shaped channels → the HTTP echo receiver, in-product channels → deterministic in-memory stores.
3Contract assertionYes (consumes Phase 2)Per channel: delivery observed, the rendered envelope carries the alert’s key fields (vulnerability id / artifact / severity), channel identity matches the etc/plugins/notify/<channel>/plugin.json manifest, and the channel-specific envelope shape validates.
4Evidence emissionNoWrites notify-coverage-summary.json + per-channel sub-reports under docs/qa/feature-checks/runs/notify/channel-coverage/<runId>/.

The harness exits non-zero iff any channel fails — a connector test fails (Phase 1), a delivery is not observed (Phase 2), or a payload contract fails (Phase 3). A deep envelope-shape miss downgrades a channel to warning (still exits 0) — delivery + key-fields + channel-identity are the gating assertions.

Channel coverage today

Email / Slack / Teams / Webhook / Discord / Telegram / OpsGenie / PagerDuty / InApp / InAppInbox have connector libraries + test projects. Cli has no standalone connector project; it is a read-surface over the InAppInbox connector, so Phase 1 records it skipped (connector-absent is not a failure) and Phase 2 routes it through the InAppInbox connector.

Webhook-shaped connectors that require HTTPS endpoints (Slack, Teams, Discord) are configured with deterministic synthetic HTTPS endpoints for connector validation. The harness’ connector-backed HTTP transport then posts the connector payload to the local HTTP echo receiver, keeping the run offline and deterministic while still exercising each connector’s DeliverAsync path.

2. Quick start (local, full 4-phase run)

2.1 Build the harness + the connector test projects

dotnet build src/E2E/StellaOps.E2E.NotifyChannelCoverage/StellaOps.E2E.NotifyChannelCoverage.csproj

# Phase 1 needs the connector test DLLs built:
for p in Email Slack Teams Webhook Discord Telegram OpsGenie PagerDuty InApp InAppInbox; do
  dotnet build "src/Notify/__Tests/StellaOps.Notify.Connectors.$p.Tests/StellaOps.Notify.Connectors.$p.Tests.csproj"
done

2.2 Start the mock receivers

docker compose -f devops/compose/docker-compose.e2e-notify-coverage.yml up -d

# Confirm both receivers are up:
curl -fsS http://localhost:8025/                 # Mailpit web UI / search API
curl -fsS http://localhost:8099/health           # notify-echo-receiver

2.3 Run the harness

dotnet exec \
  src/E2E/StellaOps.E2E.NotifyChannelCoverage/bin/Debug/net10.0/StellaOps.E2E.NotifyChannelCoverage.dll \
  --phases 1,2,3,4 \
  --mailpit-base-url http://localhost:8025 \
  --mailpit-smtp-port 1025 \
  --echo-base-url    http://localhost:8099 \
  --run-id "$(date -u +%Y%m%dT%H%M%SZ)"

echo "exit code: $?"   # 0 = all channels passed

2.4 Inspect the evidence

docs/qa/feature-checks/runs/notify/channel-coverage/<runId>/
  notify-coverage-summary.json        the top-level summary
  <channel-id>/
    phase1-test-output.txt            raw connector test-runner output
    phase1-result.json                parsed Phase 1 result
    phase2-delivery.json              Phase 2 delivery outcome + rendered envelope
    phase3-contract.json              Phase 3 contract result + notes

2.5 Tear down

docker compose -f devops/compose/docker-compose.e2e-notify-coverage.yml down --volumes --remove-orphans

3. CLI reference

dotnet exec StellaOps.E2E.NotifyChannelCoverage.dll [options]

  --phases 1,2,3,4         Phases to run (default: all). Subsets allowed;
                           --phases 1 needs no live receivers.
  --run-id <id>            Run identifier (default: UTC timestamp).
  --plugins-root <path>    Notify channel plugin manifests root
                           (default: <repo>/etc/plugins/notify).
  --mailpit-base-url <url> Mailpit HTTP API base URL
                           (default: http://localhost:8025).
  --mailpit-smtp-port <port>
                           Mailpit SMTP host port
                           (default: 1025).
  --echo-base-url <url>    HTTP echo receiver base URL
                           (default: http://localhost:8099).
  --output-root <path>     Root the run directory is written under
                           (default: repo root).
  --repo-root <path>       Repository-root override (auto-detected when omitted).
  --help, -h               Show usage.

--phases 1 is the fast CI gate — it runs the connector test projects only and needs no Docker / receivers. The full --phases 1,2,3,4 run needs the mock-receiver compose up.

4. notify-coverage-summary.json schema

{
  "runId": "20260514T093015Z",
  "generatedAt": "2026-05-14T09:30:15Z",
  "phases": {
    "connectorTestRunner": { "status": "passed", "detail": { ... } },
    "channelDelivery":     { "status": "passed", "detail": { ... } },
    "contractAssertion":   { "status": "passed", "detail": { ... } }
  },
  "channels": [
    {
      "id": "discord",
      "connector": "(none)",
      "phase1Status": "skipped",
      "phase3Status": "passed",
      "deliveryObserved": true,
      "payloadShapeValid": true
    }
  ],
  "overallStatus": "passed"
}

Field order is stable (records serialize in declaration order) and channels are ordinal-sorted by id, so two runs with the same inputs produce byte-comparable JSON.

5. CI lane

.gitea/workflows/e2e-notify-coverage.yml:

Local compose defaults remain 1025, 8025, and 8099. CI overrides the host ports with NOTIFY_COVERAGE_MAILPIT_SMTP_PORT, NOTIFY_COVERAGE_MAILPIT_HTTP_PORT, and NOTIFY_COVERAGE_ECHO_HTTP_PORT so a stale or concurrent run cannot collide with the default listener ports.

6. Troubleshooting

SymptomCauseFix
Phase 1: test assembly not foundThe connector test project was not built.dotnet build the four StellaOps.Notify.Connectors.*.Tests projects first (see §2.1).
Phase 1: cli is skippedCli has no standalone connector library; it reads InAppInbox entries.Expected — connector-absent is not a failure.
Phase 2: echo transport errorThe HTTP echo receiver is not reachable.Confirm curl http://localhost:8099/health; check docker compose ... ps.
Phase 2: SMTP transport errorMailpit is not reachable on :1025.Confirm the mailpit service is up; check the port mapping.
Docker says Bind for 0.0.0.0:1025 failed: port is already allocatedA stale local service or another compose project owns the default SMTP port.Use NOTIFY_COVERAGE_MAILPIT_SMTP_PORT=<free-port> when starting compose and pass the same value as --mailpit-smtp-port; CI does this automatically per run.
Phase 2: message sent but Mailpit search returned no matching messageThe Mailpit search API URL is wrong, or Mailpit dropped the message.Confirm --mailpit-base-url points at Mailpit’s HTTP port (:8025), not SMTP.
Phase 3: manifest id ... != catalogued plugin idThe etc/plugins/notify/<channel>/plugin.json id drifted from the catalog.Reconcile NotifyChannelCatalog with the committed manifest.
Phase 3: a channel is warningThe deep envelope-shape sub-check did not hold (non-gating).Inspect phase3-contract.json notes; the channel still passed the gating assertions.