Notify NTF-9 — attributable end-to-end delivery forcing function

Status: prepared, not executed. Run only inside the NTF-9 maintenance window, after the own-database, tenant-catalog, delivery-pipeline and web repoint gates are green. This procedure does not authorize the window.

Sprint: SPRINT_20260722_015, NTF-9.

Why a count delta is not enough

The predecessor remains available for rollback while notify-worker is proved. Both roles can therefore be members of the same durable queue consumer group. An event followed by notify.deliveries N -> N+1 proves that a consumer ran; without attribution, it does not prove the consolidated worker ran.

Rule-generated deliveries now carry the protected metadata value consumer.role. The delivery runtime derives it from the host application:

Host applicationDurable value
StellaOps.Notifier.Workernotifier-worker
StellaOps.Notify.Workernotify-worker

Any other application name or worker-id prefix fails before event handlers, rule evaluation or delivery persistence. Rule action metadata cannot override the value. The NTF-9 witness is valid only when the exact event row is delivered and carries consumer.role=notify-worker.

This probe deliberately uses InAppInbox, an internal channel, so it proves worker ownership without requiring external reachability. It does not prove OK-5’s sealed SMTP forcing case. In the active worker graph Email is external and the resolved SMTP host/port is checked immediately before every connect, including retries and FederationBundle/CryptoProvider special sends. The separately bundled connector SMTP transport has no production dispatch caller and must not be cited as a live path. A future connector-delivery activation requires its own replicated-posture guard and forcing evidence.

Preconditions — every item is a stop/go gate

  1. notify-web and notify-worker are built from a commit containing ce3ed472dcd20f662ddbf53e15b7df1dd33de9d5 (the dispatch-time SMTP and unique Eventing/Catalog migration guards), contain the consumer.role witness, and point at the same stellaops_notify database.

  2. The copy/converge parity and NOBYPASSRLS soak are green; the Authority tenant-catalog checkpoint and Platform environment_state checkpoint both exist, and the staged tenant-replica and environment-state-replica flags are on. A delivery worker with only the tenant replica is an activation NO-GO.

  3. notify-worker is healthy and its delivery pipeline is registered. Health alone is not acceptance.

  4. An operator token carrying notify.operator for the selected tenant is available through an environment variable. Never put it in shell history.

  5. The PostgreSQL operator command below reaches stellaops_notify through an operator identity. Do not grant the runtime role BYPASSRLS for this proof.

  6. Stop, but do not remove, stellaops-notifier-worker immediately before the probe event. Confirm it is not running. Its image and configuration remain the rollback path:

    docker stop stellaops-notifier-worker
docker inspect stellaops-notifier-worker --format '{{.State.Running}}'
# Required output: false

    On any failure, restart it with docker start stellaops-notifier-worker before leaving the window rollback path.

1. Capture the immutable baseline

Set local variables without printing the token:

$notifyBase = 'https://notify.stella-ops.local'
$tenant = 'default'
$postgresContainer = 'stellaops-postgres'
$database = 'stellaops_notify'
$databaseOperator = 'stellaops'

if ([string]::IsNullOrWhiteSpace($env:STELLAOPS_NTF9_OPERATOR_TOKEN)) {
    throw 'STELLAOPS_NTF9_OPERATOR_TOKEN is required.'
}
if ($tenant -notmatch '^[a-z0-9][a-z0-9-]{0,62}$') {
    throw 'Tenant must be a canonical slug.'
}

function Invoke-NotifySql([string] $sql) {
    $lines = @(& docker exec $postgresContainer psql -X -v ON_ERROR_STOP=1 `
        -U $databaseOperator -d $database -Atc $sql)
    if ($LASTEXITCODE -ne 0) { throw 'Notify witness SQL failed.' }
    return $lines
}

$baseline = [int](Invoke-NotifySql `
    "SELECT count(*)::text FROM notify.deliveries WHERE tenant_id = '$tenant';")
"baseline=$baseline"

The copied estate was measured at 1; if the live baseline differs, compare it with the copy/parity record before proceeding. Do not overwrite the recorded number. The invariant for this run is the captured N -> N+1, not a hard-coded historical count.

2. Seed one bounded in-app channel, template and rule through the API

This is QA scaffolding, never migration/seed data. Unique IDs make retries non-destructive and let cleanup target only this run.

$run = [guid]::NewGuid().ToString('N')
$eventId = [guid]::NewGuid()
$eventKind = 'release.ntf9.delivery.probe'
$channelId = "ntf9-channel-$run"
$templateId = "ntf9-template-$run"
$ruleId = "ntf9-rule-$run"
$headers = @{
    Authorization = "Bearer $($env:STELLAOPS_NTF9_OPERATOR_TOKEN)"
    'Content-Type' = 'application/json'
}

$channel = @{
    name = "NTF-9 in-app probe $run"
    type = 'InAppInbox'
    purpose = 'general'
    enabled = $true
    config = @{
        secretRef = "legacy://ntf9/$run"
        target = "ntf9-probe-user-$run"
    }
} | ConvertTo-Json -Depth 8
Invoke-RestMethod -Method Put -SkipCertificateCheck -Headers $headers `
    -Uri "$notifyBase/api/v2/notify/channels/$channelId" -Body $channel | Out-Null

$template = @{
    templateId = $templateId
    key = "ntf9.delivery.probe.$run"
    channelType = 'InAppInbox'
    locale = 'en'
    body = "NTF-9 delivery forcing witness $run"
    renderMode = 'PlainText'
    format = 'InAppInbox'
} | ConvertTo-Json -Depth 8
Invoke-RestMethod -Method Post -SkipCertificateCheck -Headers $headers `
    -Uri "$notifyBase/api/v2/notify/templates" -Body $template | Out-Null

$rule = @{
    ruleId = $ruleId
    name = "NTF-9 exact-event probe $run"
    enabled = $true
    match = @{ eventKinds = @($eventKind) }
    actions = @(@{
        actionId = "ntf9-action-$run"
        channel = $channelId
        template = $templateId
        enabled = $true
        metadata = @{ 'ntf9.run' = $run }
    })
} | ConvertTo-Json -Depth 12
Invoke-RestMethod -Method Post -SkipCertificateCheck -Headers $headers `
    -Uri "$notifyBase/api/v2/notify/rules" -Body $rule | Out-Null

Use InAppInbox: a delivered result is written only after the durable inbox write succeeds, and the probe has no external-network side effect.

3. Publish one event directly to the service alias

This follows the real producer path; do not use the gateway for this step.

$event = @{
    eventId = $eventId
    kind = $eventKind
    tenant = $tenant
    timestamp = (Get-Date).ToUniversalTime().ToString('o')
    correlationId = "ntf9-$run"
    payload = @{ marker = $run }
    attributes = @{ 'ntf9.run' = $run }
} | ConvertTo-Json -Depth 10

Invoke-RestMethod -Method Post -SkipCertificateCheck -Headers $headers `
    -Uri "$notifyBase/api/v1/events/releaseorchestrator" -Body $event | Out-Null

An HTTP 202 is queue admission only. It is not a pass.

4. Require the exact durable witness

Poll for at most 60 seconds. The query is scoped by tenant, source eventId, run marker and successor role; a different delivery cannot satisfy it.

$deadline = [DateTimeOffset]::UtcNow.AddSeconds(60)
$witness = @()
do {
    $witness = @(Invoke-NotifySql @"
SELECT status::text || E'\t'
    || COALESCE(event_payload->'metadata'->>'consumer.role', '') || E'\t'
    || COALESCE(event_payload->'metadata'->>'ntf9.run', '')
FROM notify.deliveries
WHERE tenant_id = '$tenant'
  AND event_payload->>'eventId' = '$($eventId.ToString('D'))'
  AND event_payload->'metadata'->>'ntf9.run' = '$run'
ORDER BY created_at;
"@)
    if ($witness.Count -eq 1 -and $witness[0] -eq "delivered`tnotify-worker`t$run") {
        break
    }
    Start-Sleep -Seconds 2
} while ([DateTimeOffset]::UtcNow -lt $deadline)

if ($witness.Count -ne 1) {
    throw "Expected exactly one exact-event delivery witness; observed $($witness.Count)."
}
if ($witness[0] -cne "delivered`tnotify-worker`t$run") {
    throw "Delivery witness was not delivered by notify-worker: $($witness[0])"
}

$after = [int](Invoke-NotifySql `
    "SELECT count(*)::text FROM notify.deliveries WHERE tenant_id = '$tenant';")
if ($after -ne ($baseline + 1)) {
    throw "Delivery count did not move by exactly one: $baseline -> $after."
}

"PASS event=$eventId count=$baseline->$after consumer=notify-worker status=delivered run=$run"

The comparison is deliberately case-sensitive. PostgreSQL JSON keys and values are case-sensitive, and accepting a normalized or missing role would turn old-image evidence into a false pass.

5. Cleanup and retained evidence

Delete only the three configuration records created by this run. This cleanup is mandatory on both PASS and NO-GO; otherwise an enabled rule from a failed run would match the next probe and manufacture a multiple-row failure.

$cleanupErrors = [System.Collections.Generic.List[string]]::new()
@(
    "$notifyBase/api/v2/notify/rules/$ruleId"
    "$notifyBase/api/v2/notify/templates/$templateId"
    "$notifyBase/api/v2/notify/channels/$channelId"
) | ForEach-Object {
    $resourceUri = $_
    try {
        $response = Invoke-WebRequest -Method Delete -SkipCertificateCheck `
            -SkipHttpErrorCheck -Headers $headers -Uri $resourceUri
        if ($response.StatusCode -notin 204, 404) {
            $cleanupErrors.Add("${resourceUri}: HTTP $($response.StatusCode)")
        }
    }
    catch {
        $cleanupErrors.Add("${resourceUri}: $($_.Exception.Message)")
    }
}
if ($cleanupErrors.Count -gt 0) {
    throw "NTF-9 probe cleanup failed: $($cleanupErrors -join '; ')"
}

Retain the exact delivery and inbox rows as the append-audit witness. Record: baseline, final count, event ID, run ID, delivery status, consumer.role, image digest and the stop/start state of the predecessor. Never put the bearer token, channel secret or connection password in the evidence.

Fail-closed outcomes

Any of these is NO-GO and triggers the predecessor restart/rollback path:

Health, logs, queue acknowledgement and a count delta without the exact role are supporting evidence only; none can close NTF-9.