Notify — building and deploying the staged consolidated worker (NTF-8)

Sprint: SPRINT_20260722_015 NTF-8 (S8 DevOps). Sibling precedents: jobengine-deploy-job8-staged-stack.md, graph-deploy-gra8-staged-stack.md, advisoryai-deploy-aai8-staged-stack.md.

Staged artifacts this page covers:

ArtifactPath
Overlaydevops/compose/docker-compose.notify.yml
Publish keynotify-worker (devops/docker/build-service-publish.sh:292)
Connection variableSTELLAOPS_POSTGRES_NOTIFY_CONNECTION
Env templatedevops/compose/env/stellaops.env.example

What the staged stack serves today (read this before rehearsing)

One role, not two. The overlay stages notify-worker only. notify-web already exists, live, at its target name in docker-compose.stella-services.yml — staging a second definition would give one live service two. Its source and Compose connection repoint onto stellaops_notify are now prepared by the F-NTF9-1 option-1 decision, but the copy/recreate/soak remains the NTF-9 window’s job. Editing the repository does not change the running container.

The staged worker DELIVERS NOTHING, and that is the honest state rather than a defect. It converges the schema, serves health, and answers GET /doctor/notify-web/checks. F-NTF9-1 option 1 keeps the existing DAL and registers its runtime repositories against the same own-database connection in both target roles; the legacy migration lineage is not registered there. The predecessor-equivalent StellaOps.Notify.Delivery composition root is now present in source, but Notify:DeliveryPipeline:Enabled is absent/default-false and the staged overlay does not set it. In that state the host registers one explicit disabled witness and neither queue consumer. Source presence is not activation evidence.

notifier-worker is NOT retired here and cannot be. That container is the live CONSUMER of the notification event stream, so retiring it before the successor runs the pipeline stops event processing. Its compose key, its publish key, and the notifier-worker|notify boundary-register pin therefore all survive to the window — see “What NTF-9 and NTF-10 still owe” below.

Corrected 2026-08-18 (sprint 015 F-NTF9-4) — it is not “the container actually delivering notifications”, and the difference decides how you prove the cutover. The earlier claim rested on stellaops-notifier-worker being up and healthy, which is a health probe standing in for a delivery claim. Measured: notify.deliveries holds exactly one row, dated 2026-05-30 (every other monthly partition zero) and notify.rules holds zero rows. The container is emphatically alive — 30,026 log lines in 24 h — but every processing line reads Processed event <id> for tenant default; created 0 deliveries, because with no rule configured nothing can match. It consumes; it does not deliver.

Consequence for the window: NTF-10’s precondition — “notify-worker running AND delivering, not merely healthy” — cannot be satisfied by observing ambient traffic, because this estate produces none. The forcing function has to be CONSTRUCTED: seed a channel and a matching rule (there are none), publish an event on the stream the worker already consumes, and assert a NEW row in notify.deliveries. Prove it the way the only executed sibling window did (SPRINT_20260722_020 W3-01 timeline, 96,419 → 96,421): as a count delta against a recorded baseline, which here is 1, so the expected observable is notify.deliveries going 1 → 2. A delta alone is still ambiguous while the predecessor is retained for rollback: the exact row must also be delivered with consumer.role=notify-worker. Follow notify-ntf9-delivery-forcing-function.md for the bounded seed, publish, exact-row assertion and cleanup. The seed data is QA scaffolding and must never reach a migration (§2.11).

Build the image

bash devops/docker/build-service-publish.sh notify-worker

The map’s third field is a trap worth understanding before you read it as a bug. The notify-worker entry ends stellaops/notifier-worker:dev, i.e. the PREDECESSOR’s image name. That field is the legacy-mode-only base; the publish helper derives the OUTPUT tag from the KEY, so this build produces stellaops/notify-worker:dev and cannot overwrite the live predecessor image. Verified at NTF-8. Do not “fix” the third field.

Do not make this host Microsoft.NET.Sdk.Web. It was, briefly, copied from findings-worker; src/Directory.Build.props:190 injects StellaOps.Router.AspNet into every project where UsingMicrosoftNETSdkWeb is true, which dragged Router, Microservice, Plugin, Cryptography, Auth, Catalog.Replication and AirGap.Policy into a worker that serves no domain route — 29 published StellaOps assemblies instead of 7. It is Microsoft.NET.Sdk + an explicit Microsoft.AspNetCore.App FrameworkReference, the shape StellaOps.Worker.Health itself takes.

The build-boundary report cannot see this. It walks csproj ProjectReference items, so an MSBuild-CONDITIONAL injection is invisible to it — it read “7 projects in closure, 0 violation pairs” the entire time the publish emitted 29. The check that actually catches it is comparing the published assembly count against the report’s closure count. This is an estate-wide gate property, not a Notify one: findings-worker, jobengine-worker and evidence-worker are all Sdk.Web with understated closures.

Bring up an isolated scratch stack

Never against the live network before the window. The overlay stages no PostgreSQL cluster — the family owns the logical database stellaops_notify + the notify role on the SHARED installation server (ADR-039 P1). A rehearsal supplies its own throwaway server and provisions with the same script the NTF-9 window uses, so the rehearsal exercises the window’s provisioning step.

docker network create ntfscratch
docker run -d --name ntfscratch-pg --network ntfscratch \
  -e POSTGRES_USER=stellaops -e POSTGRES_PASSWORD=<scratch-password> \
  <audited digest-pinned postgres image>

PGPASSWORD_SERVICE=<scratch-password> \
  bash tools/scripts/deploy/postgres/provision-service-database.sh \
    stellaops_notify notify --container ntfscratch-pg

INSTANCE_SUFFIX=-ntfscratch \
STELLAOPS_NOTIFY_NETWORK=ntfscratch \
STELLAOPS_POSTGRES_NOTIFY_CONNECTION='Host=ntfscratch-pg;Port=5432;Database=stellaops_notify;Username=notify;Password=<scratch-password>;Maximum Pool Size=16' \
  docker compose -p ntfscratch --env-file /dev/null \
    -f devops/compose/docker-compose.notify.yml up -d

Three things in that invocation are load-bearing:

The password appears twice (creating the role, and inside the connection string). Compose does not expand ${VAR} inside env-file values, so they are kept in sync by hand, like every other service in the estate.

STELLAOPS_POSTGRES_NOTIFY_CONNECTION is required with no fallback, in the compose file and in the service. A Notify host that starts against an unintended database is worse than one that does not start.

Verifying convergence — a forcing function, not “the container is up”

# 1. The service converged ITS OWN database from empty. The provisioning script
#    creates the database and role and nothing else (AGENTS.md §2.7) — an empty
#    database is the contract.
docker exec ntfscratch-pg psql -U stellaops -d stellaops_notify -c \
  "SELECT schemaname, count(*) FROM pg_tables
    WHERE schemaname IN ('notify','eventing') GROUP BY 1 ORDER BY 1;"
#    Measured on the NTF-8 scratch boot: notify 33 (the 32-table design set plus
#    the migration ledger — no stray table) and eventing 6.

# 2. §2.11: every owned table is EMPTY on a fresh converge, and the ledger holds
#    its baseline once.

# 3. Health. The hardened base ships NEITHER curl NOR wget, so probe over
#    /dev/tcp, the same fallback devops/docker/healthcheck.sh uses.
docker exec stellaops-notify-worker-ntfscratch bash -c \
  'exec 3<>/dev/tcp/127.0.0.1/8080 && printf "GET /health/readiness HTTP/1.0\r\n\r\n" >&3 && head -1 <&3'

# 4. The doctor surface: FIVE checks — the three standard ones plus this family's
#    two, doctor.notify.rls-posture and
#    doctor.notify.deliveries.partition-coverage.

NTF-8 measured this healthy on the first try with RestartCount 0. Tear down with docker compose -p ntfscratch down -v and confirm no scratch container, network or volume survives.

Redeploy trap (every time, not only the first)

A same-tag image does not swap on up -d. After rebuilding:

docker compose <the service's OWN -f list> up -d --force-recreate --no-deps notify-worker

Recreate with the service’s own config_files label set — the estate’s labels are heterogeneous, and recreating with the wrong -f list rewrites the service. Read it from the container:

docker inspect stellaops-notify-worker \
  --format '{{index .Config.Labels "com.docker.compose.project.config_files"}}'

Expect a HELLO re-send over Valkey after any swap; a gateway 503 "no instances available" immediately afterwards is fixed by one docker restart.

Doctor registration — deliberately absent on this role

There are no Doctor__Registration__* variables in the overlay, and adding them is a defect (D-NTF7-1). Registration is not wired on the worker at all: the checks are served locally, and the Platform capability handshake lands on notify-web at NTF-9, where the Authority client closure already exists. Adding the variables here advertises a capability the host does not have.

Remaining blockers the window must clear before this host can do production work

What NTF-9 and NTF-10 still owe (recorded here so it is not lost)

NIS2 ledger hand-off gate and recovery

Notify:Nis2LedgerOutbox:Enabled is mapped by Compose but defaults false. Every retention/lease value is blank in repository configuration, so enabling the gate without an explicit window contract fails startup. Disabled activation returns HTTP 503 for incident writes. The two-tenant live owner proof passed on 2026-09-11 before retiring the synchronous Findings writer; do not restore that path.

Before activation, all of the following must be recorded and green:

  1. Findings owns durable consumer findings-nis2-incident-ledger-v1 for the stable logical stream notify.nis2-incident-ledger, schema nis2.incident.ledger-handoff.v1, with payload EventId as its tenant-scoped ledger idempotency key. Notify persists each tenant in notify.nis2-incident-ledger.tenant.<lowercase-utf8-hex>; the owner routes derive the tenant only from the authenticated context, echo it in every response, and never accept a tenant in the route, query or request body. Findings must require that echo to equal its configured control/token tenant. It pulls the direct Notify owner route GET /api/v1/notify/internal/nis2-incident-ledger/events and reports only its committed cursor to POST /api/v1/notify/internal/nis2-incident-ledger/consumers/findings-nis2-incident-ledger-v1.
  2. The consumer’s inbox/checkpoint/lease and producer registration are physical-stream scoped. The producer retention declaration expands only the exact delimiter-terminated notify.nis2-incident-ledger.tenant. family and prevents eventing.outbox pruning ahead of consumption; a near-prefix must remain untouched.
  3. notify-web is pointed at the copied/converged stellaops_notify; the ordinary service role remains NOBYPASSRLS and its tenant-attributed connection path is proven.
  4. A bounded incident transition returns ledgerStatus=Queued, and read-only checks show both the matching notify.incident_report_timeline_states row and exactly one outbox payload whose payload.event.eventId equals the request event id. The same transaction must also contain one notify.nis2_incident_ledger_handoffs row for that tenant/event with the envelope receipt and payload SHA. After one deliberate replay, the outbox must still contain exactly one committed envelope for that payload id, the returned receipt must be unchanged, and Findings must contain exactly one corresponding ledger event. P6 inbox envelope-id dedupe is not sufficient. Replays retain the original metadata.eventId and metadata.occurredAt; omitted values are assigned by the receiving host. payload_sha256 binds the complete first wire payload, including its original server recordedAt. The separate request_sha256 binds the same content with only that server timestamp normalized to the Unix epoch. Advancing the server clock or restarting the owner must not change the original receipt, payload, or timeline. Changing caller event content under the same identity must return conflict.
  5. Before starting an upgraded owner over legacy data, apply 004_nis2_tenant_stream_partition.sql under the recorded writer freeze. It must rekey the original global rows and matching hand-off receipts transactionally while retaining the legacy stream state/registrations. different epoch is a hard stop: restore the predecessor image and configuration against the intact source and follow this runbook’s rollback section. Never accept the migration ledger row or activate either flag after that failure.
  6. Before enabling this previously inactive producer, record an empty notify.nis2_incident_ledger_handoffs census under the ordinary owner for every Authority tenant, and retain the writer fence through startup migration 005_nis2_request_identity.sql. It adds the mandatory request fingerprint without rewriting any receipt or applied migration. Unexpected existing receipts make startup fail transactionally; investigate them before activation. Never fabricate a fingerprint or delete receipts to make this gate pass.

The durable activation surface is split deliberately. On findings-worker, set FINDINGS_NIS2_LEDGER_CONSUMER_ENABLED=true plus the explicit owner origin, control tenant, client-credentials authority/token endpoint/secret, poll/page/ lease/response/timeout values; its cache root is the Findings-owned persistent volume at /app/data/findings-advisory-corpus/nis2-ledger. On notify-web, set NOTIFY_NIS2_LEDGER_OUTBOX_ENABLED=true, a positive NOTIFY_NIS2_LEDGER_REMOTE_CONSUMER_LEASE, and every NOTIFY_NIS2_OUTBOX_RETENTION_* value. The retention window must exceed the remote-consumer lease and RequirePublished must be explicit false.

Order is consumer-first: deploy findings-worker, register it, record the owner head and, when a head exists, wait until its durable checkpoint and producer report equal that head. For an empty source record null/zero registration as described below. Then recreate only notify-web with the producer gate true. On any red, disable the producer and NIS2 consumer flags while preserving the worker’s other projectors. Disabled incident writes fail explicitly. If the Notify database repoint itself is red, recreate the recorded predecessor image/config against the intact source database. Do not reverse-copy target rows and do not drop either target database; they remain forensic/rollback evidence.

The live tenant-isolation forcing proof uses both recorded Authority assignments (default and e2e-lab) without changing the logical wire name. Seed one uniquely marked owner envelope per tenant. Each authenticated catch-up must return only its own marker and tenant echo; each tenant’s reported cursor and Findings checkpoint must advance independently from sequence 1; replaying one tenant must not change the other’s checkpoint. A cross-tenant payload, response echo, report or lease is a hard red before inbox mutation. Roll back both activation flags immediately on any cross-read, cross-ack, sequence interference, or tenant mismatch.

For an isolated NIS2-only proof instance, keep unrelated projectors disabled. For the live worker, preserve the recorded advisory, SBOM and Scanner activation settings and identities. The NIS2 owner HTTP pipeline has its own Auth.Client options and token cache; it must not replace the advisory client’s root identity. The current estate already runs advisory and Scanner consumers, so disabling them is not a prerequisite for this hand-off.

An empty producer has no stream epoch yet. Record its authenticated consumer registration at null epoch/sequence zero and its empty catch-up response; do not invent a destination checkpoint. The first approved owner envelope establishes the real epoch and must then appear in both the durable Findings checkpoint and the producer’s reported cursor before the live forcing proof can pass.

Any missing timeline row, missing/duplicate outbox payload, response other than Queued, absent Findings ledger event, or checkpoint that does not advance is NO-GO: set the gate false and retain the recorded images/configuration and all committed records for recovery. The initial proof passed before NTF-9 removed the legacy adapter, Findings implementation reference and notify-web|findings pin. Source preparation alone is not liveness or cutover evidence; the final image must replay the retained proof identities and preserve the same receipts and cursors.

NTF-8’s task text says the notifier-worker key “becomes” notify-worker. It cannot, at S8 — see the top of this page. The retirement is therefore a window step, and it has three parts that must land TOGETHER or the boundary check fails on key drift:

  1. Delete the notifier-worker key from devops/docker/build-service-publish.sh.
  2. Delete its entry from docs/architecture/build-boundary/ownership-manifest.json (these two must stay in lockstep).
  3. Delete the notifier-worker|notify register pin, and prove -Check green on it (this is what closes the D-NTF4-5 deviation).

F-NTF9-1 option 1 deliberately keeps the existing DAL, so NTF-10 does not delete StellaOps.Notify.Persistence or rename StellaOps.Notify.Persistence.Consolidated onto an occupied name. The former remains the runtime repository home; the latter remains the target migration authority with its explicit D12 exception. NTF-10 still retires the predecessor, old central-migrator path, old namespaces, and obsolete deployable artifacts.

Rollback

Nothing to roll back before the window: the overlay is outside the default config_files list, so nothing running changes until it is explicitly composed in. If a rehearsal is in flight, docker compose -p ntfscratch down -v removes it entirely. During the window, rollback is dropping the overlay from the -f list and force-recreating; notifier-worker keeps consuming the event stream throughout, which is precisely why it is not retired first. Ambient delivery is not available as evidence because the live estate currently has no matching rules; use the constructed, consumer-attributed forcing function in notify-ntf9-delivery-forcing-function.md.