Shared-catalog replication contract (catalog-replication/v1)

Status: v1, 2026-08-02 (SPRINT_20260722_027 SCR-1/SCR-2, X14). Runtime: src/__Libraries/StellaOps.Catalog.Replication. Producers: Platform (actor_identity, 2026-08-02) and Authority (tenants, SPRINT_20260722_016, 2026-08-03).

1. Why

Every service that resolves tenants or actor identities historically read shared.tenants / shared.actor_identity across a database boundary (X14 in the ownership register). Under ADR-039 those reads die: each consumer converges a LOCAL replica in its OWN database from producer events plus a bootstrap pull, and reads never leave the host again — which is also what preserves the offline posture (a producer outage stops replication from advancing; it never stops a resolver from answering).

2. Roles

Host-owned migration resources

StellaOps.Catalog.Persistence embeds the three catalog migrations; StellaOps.Eventing.Persistence embeds the six reliability migrations. AddCatalogReplication and AddEventingReliability select those resource assemblies through typed markers. The runtime libraries contain no migration SQL. The persistent module names remain Catalog.Replication and Eventing.Reliability, and each host applies the unchanged SQL leaf names in its own database under catalog_replica.schema_migrations and eventing.schema_migrations.

This changes no connection, tenant, stream, retention or activation setting. Source verified against 85206304e9 (EST-9). Live-owner restart verification is recorded separately by the Program sprint. Tests that invoke the migration runner directly must select the persistence marker, not the assembly of a runtime store. SharedMigrationStartupTests exercises the real DI registrations against fresh PostgreSQL, verifies all nine ledger names, and compares every ledger field after a second startup. Run it with:

dotnet test src/__Libraries/__Tests/StellaOps.Catalog.Replication.Tests/StellaOps.Catalog.Replication.Tests.csproj

• Producer owns the catalog table (source of truth, unchanged) and, in the SAME transaction as every domain write, appends a change event to its own P6 outbox (CatalogChangeProducer). It serves the pull feed (MapCatalogChangeFeed) from that outbox under the catalog:replicate MACHINE scope, restricted to an explicit servable-catalog allowlist.
• Consumer drains the feed (CatalogReplicationConsumer: durable checkpoint, inbox dedup, per-event destination transaction, producer-epoch re-sync) into catalog_replica.replica_rows in its OWN database, and resolves from there (CatalogReplicaStore). The replica table’s P13 class is derived-rebuildable (003_catalog_replica_retention_class.sql): discarding a catalog’s rows together with its consumer checkpoint re-converges them through the same pull from sequence 0, and no row is ever removed (deletes are tombstones).

3. Wire shape

GET {producer}/catalog-changes/{catalog}?afterSeq=N&limit=M     scope: catalog:replicate

{ "catalog": "actor_identity",
  "streamEpoch": 1754127339123,        // null = producer has never emitted (NOT epoch 0)
  "headSeq": 42,                        // lag = headSeq - checkpoint
  "changes": [ {
      "eventId": "…", "streamEpoch": …, "seq": …,
      "key": "{tenantId}:{actorRef}",
      "version": 3,                     // monotonic per key (row_version, never a timestamp)
      "payload": { …the versioned row… },
      "deleted": false,
      "occurredAt": "2026-08-02T…"
  } ] }

Rules (all enforced by the runtime, all covered by CatalogReplicationTests/CatalogWireRoundTripTests):

1. Versioned application. A change applies only when version is newer than the stored row’s — duplicates and out-of-order deliveries are no-ops by construction. Versions come from a per-row counter bumped by the producer’s write path; timestamps are not versions (same-tick writes would tie, and a tie is a lost update on a replica).
2. Deletes are tombstones. A replica never removes the row — removal could be resurrected by an out-of-order older event.
3. Epoch discipline. streamEpoch changes when the producer is rebuilt/restored; the consumer discards its checkpoint and re-syncs. The epoch comparison runs BEFORE the empty-page return (an empty batch is not evidence that the checkpoint is still valid).
4. Payloads carry the full versioned row, minimized. Catalogs are small reference data; shipping the row lets a replica converge without calling back.

4. Event types (DC-06)

Tenants semantics. authority.tenants.row_version (migration 023) is the monotonic version; a BEFORE UPDATE trigger bumps it for every writer, so no write path can forget. Suspension is a versioned status update (the tenant still exists; resolvers must see suspended), while a hard delete is a tombstone (the tenant is gone; replicas stop resolving it). Tenants created by migrations/seeds predate the producer and have no catalog history — the startup backfill (TenantCatalogBackfill, ordered after the authority + eventing migrations) emits them once, tombstones catalog keys whose row vanished behind the producer’s back, and is silent at steady state. The feed rides Authority itself: GET {authority}/catalog-changes/tenants under the same catalog:replicate machine scope.

Erasure semantics (GDPR, actor-identity). An erase is a versioned update, never a tombstone: the payload ships the already-nulled row with erased: true, so every replica converges to the erased state with zero PII on the wire. A tombstone would be wrong — a tombstoned replica row projects as unknown actor, and erased and unknown are DIFFERENT shapes in the three-shape identity projection. The SCR-4 offboarding cascade builds on this: erasure propagates to replicas through the ordinary replication path, no side channel.

5. Tenant lifecycle events (SCR-4, v1 producer half)

Typed offboarding events ride the SAME wire shape on their own stream, tenant_lifecycle, served by Authority’s feed under the same catalog:replicate scope. Consumers treat it as an ORDERED EVENT FEED (inbox dedup on (stream, epoch, seq)), never as versioned-row upserts.

Consumer contract on tenant.deleted: dispose the tenant’s data per your own P13 retention classes, with two hard overrides — evidence legal holds (evidence_holds) block disposal of held bundles, and ledger-class stores (Findings Merkle partitions) archive-then-detach rather than delete. Per DC-06, ignore unknown event types. The consumer-side handler surface + dry-run default land with the SCR-4 library half; per-service handlers ride each service’s program (or record an explicit not-applicable).

The evidence override, as actually implemented (011 EVD-11, 2026-08-18). EvidenceTenantLifecycleHandler (src/Evidence/StellaOps.Evidence.WebService/ TenantLifecycle/) is the named legal-holds adopter, and two measured facts refine the one-line rule above rather than contradicting it:

• “Block disposal of held bundles” is weaker than what the schema already guarantees. evidence_bundles_block_mutation raises EVIDENCE_BUNDLE_IMMUTABLE on EVERY delete of a sealed bundle, held or not, so no handler can dispose one in the first place. Of the 24 tenant-scoped tables in the consolidated database NONE is derived-rebuildable today: the proof graph (proofchain.graph_nodes, graph_edges) was, until the RFC-8 retention classification (009_evidence_retention_classes.sql, 2026-09-21) made it source-of-truth because explicit node and edge writes carry data no replay reconstructs. Every other class is refused, so a tenant.deleted event currently disposes no Evidence rows and the report lists each table as kept with its class. Discovery reads the classes from the live catalog, so a table later classed derived-rebuildable is disposed without a code change.
• The override therefore applies at TENANT granularity, not per bundle. No deletable table in the family carries a bundle reference, so “skip the held bundle’s rows” has no join to express it. An active hold — released_at IS NULL and no elapsed expires_at — suspends every deletion for that tenant, which is also what the baseline’s own P13 preamble asks for (“no class licenses deleting a held bundle or anything hanging off it”): the proof graph is what makes the held evidence interpretable.

A hold with a NULL bundle_id is unattributable and WIDENS the withholding rather than being ignored.

6. Consumer checklist (per SCR-3 flag flip)

1. Register AddCatalogReplication() (homes catalog_replica in your own DB) and AddEventingReliability() (inbox/checkpoints).
2. Wire HttpCatalogChangeFeed on a named client carrying your service credentials with the catalog:replicate grant.
3. Drain on your own schedule; read through CatalogReplicaReader, not CatalogReplicaStore directly — see §7 — and never through shared.* again.
4. The producer’s table stays the source of truth; your replica is rebuildable (drop + re-drain is always safe).

7. Reading a replica: three states, not two (SCR-3)

A replica read has three distinguishable outcomes, and collapsing the last two is the defect this section exists to prevent:

CatalogReplicaReader.LookupAsync returns that tri-state. SourceUnavailable covers a catalog_replica schema that has not converged, an unreadable/ unreachable database, and — the one a row count cannot see — a replica that has never drained. The witness for that last case is the drain’s durable checkpoint or its owner-confirmed empty observation, not a row count. A null stream epoch says only that the producer has not emitted; it does not prove that a producer’s domain table is empty before backfill. Without either durable witness, absence remains SourceUnavailable.

For a never-emitted stream, the producer may include emptyCatalogConfirmed:true only after its explicitly registered owner probe confirms an empty domain catalog. Platform opts in environment_state through its own IEnvironmentStateStore; other catalogs do not acquire this behavior automatically. The HTTP reader requires a matching catalog, explicit null streamEpoch and headSeq, an explicit empty changes array and the boolean confirmation. Missing/wrong-type/duplicate fields, contradictory confirmation, non-200 responses and failed authentication never establish an observation.

The consumer stores the confirmed observation in its own catalog_replica.empty_observations table (forward migration 002). It creates no event, epoch zero, seal document or Eventing checkpoint. The observation survives producer loss. A first real event removes it in the same local transaction as the row application and real checkpoint. Existing rows/checkpoints are preserved if a later response claims a never-emitted producer. Empty observations and event writes share a short local transaction advisory lock; no lock is held over HTTP. The consumer rechecks the checkpoint after acquiring that lock and discards a response whose starting checkpoint changed while the HTTP request was in flight. The reader obtains its row/checkpoint/empty-observation answer from one local repeatable-read snapshot, so a first-event race cannot combine an old missing row with a new checkpoint to invent absence.

empty_observations is derived-rebuildable: one row per observed empty catalog, removed on the first event or rebuilt by authenticated observation after an intentional replica reset. It is not a TTL cache, and it must not expire simply because the producer is offline. The migration adds no fixture rows or new runtime DDL writer; each existing AddCatalogReplication host owns its schema migration.

Reading by payload field (the reverse direction)

CatalogReplicaReader.LookupByPayloadFieldAsync answers the mirror question — which live row carries this value in its payload? — with the same three states, the same single repeatable-read snapshot and the same never-drained witness as LookupAsync. The tenants catalog needs it because the replica is keyed by the slug while the canonical UUID lives in the payload, so a consumer holding a UUID has no key to look up.

Two rules are specific to a reverse read:

• It is not unique by construction. PRIMARY KEY (catalog, key) says nothing about payload contents, so two live rows may carry the same value. That is SourceUnavailable, never a pick-the-first: a replica in that state cannot answer correctly, and choosing one would make an inconsistent replica look like a verdict.
• The field name is a bind parameter (payload->>@field), which is what makes it safe for a generic API. It also settles the index question: an expression index on (payload->>'id') can never be matched when the key is a parameter, so none is added. The existing partial index on (catalog) WHERE deleted = FALSE restricts the scan to one catalog’s live rows, which is what does the work.

The tenants-shaped accessor over it is IStellaOpsTenantKeyResolver (ReplicaStellaOpsTenantKeyResolver), registered by the same AddStellaOpsTenantResolverReplicaOnly call as the forward seam, so a flipped host acquires both directions with no extra configuration key. It is a separate interface on purpose: widening IStellaOpsTenantResolver would make every implementation owe a reverse it has no data for.

CatalogReplicaLookup.EnsureAnswered() converts SourceUnavailable into CatalogReplicaUnavailableException, whose message names the flag, the catalog:replicate grant and the drain. Callers whose return type cannot carry a third state — IStellaOpsTenantResolver returns Guid? — must throw rather than return the empty value, because a consumer reads that empty value as an answer: a null tenant surfaces as tenant_invalid_format (HTTP 400, blaming the caller) or “does not resolve to a known tenant” (HTTP 500 naming the wrong cause). That exact conflation cost the SPRINT_20260722_007 POL-F6 cutover two rollbacks: a gate evaluation decided correctly, was attested, and then failed its decision-history write with a message about the tenant identifier while the real fault was a missing Platform-owned relation.

Bootstrap window. Until the first drain commits a checkpoint or a confirmed empty observation, a missing-row read is SourceUnavailable — correctly, since the replica genuinely cannot answer. The tenants seam narrows that window by draining once, bounded, before host startup completes (Catalog:Replication:Tenants:BootstrapTimeout, default 30 s), ordered after the startup-migration hosts by registration order. It is deliberately non-fatal: a host whose replica has already converged must start even with a dead producer (the SCR-1 kill-the-producer posture).