Shared-catalog replication contract (catalog-replication/v1)

Status: v1, 2026-08-02 (SPRINT_20260722_027 SCR-1/SCR-2, X14). Runtime: src/__Libraries/StellaOps.Catalog.Replication. Producers: Platform (actor_identity, 2026-08-02) and Authority (tenants, SPRINT_20260722_016, 2026-08-03).

1. Why

Every service that resolves tenants or actor identities historically read shared.tenants / shared.actor_identity across a database boundary (X14 in the ownership register). Under ADR-039 those reads die: each consumer converges a LOCAL replica in its OWN database from producer events plus a bootstrap pull, and reads never leave the host again — which is also what preserves the offline posture (a producer outage stops replication from advancing; it never stops a resolver from answering).

2. Roles

• Producer owns the catalog table (source of truth, unchanged) and, in the SAME transaction as every domain write, appends a change event to its own P6 outbox (CatalogChangeProducer). It serves the pull feed (MapCatalogChangeFeed) from that outbox under the catalog:replicate MACHINE scope, restricted to an explicit servable-catalog allowlist.
• Consumer drains the feed (CatalogReplicationConsumer: durable checkpoint, inbox dedup, per-event destination transaction, producer-epoch re-sync) into catalog_replica.replica_rows in its OWN database, and resolves from there (CatalogReplicaStore).

3. Wire shape

GET {producer}/catalog-changes/{catalog}?afterSeq=N&limit=M     scope: catalog:replicate

{ "catalog": "actor_identity",
  "streamEpoch": 1754127339123,        // null = producer has never emitted (NOT epoch 0)
  "headSeq": 42,                        // lag = headSeq - checkpoint
  "changes": [ {
      "eventId": "…", "streamEpoch": …, "seq": …,
      "key": "{tenantId}:{actorRef}",
      "version": 3,                     // monotonic per key (row_version, never a timestamp)
      "payload": { …the versioned row… },
      "deleted": false,
      "occurredAt": "2026-08-02T…"
  } ] }

Rules (all enforced by the runtime, all covered by CatalogReplicationTests/CatalogWireRoundTripTests):

1. Versioned application. A change applies only when version is newer than the stored row’s — duplicates and out-of-order deliveries are no-ops by construction. Versions come from a per-row counter bumped by the producer’s write path; timestamps are not versions (same-tick writes would tie, and a tie is a lost update on a replica).
2. Deletes are tombstones. A replica never removes the row — removal could be resurrected by an out-of-order older event.
3. Epoch discipline. streamEpoch changes when the producer is rebuilt/restored; the consumer discards its checkpoint and re-syncs. The epoch comparison runs BEFORE the empty-page return (an empty batch is not evidence that the checkpoint is still valid).
4. Payloads carry the full versioned row, minimized. Catalogs are small reference data; shipping the row lets a replica converge without calling back.

4. Event types (DC-06)

Tenants semantics. authority.tenants.row_version (migration 023) is the monotonic version; a BEFORE UPDATE trigger bumps it for every writer, so no write path can forget. Suspension is a versioned status update (the tenant still exists; resolvers must see suspended), while a hard delete is a tombstone (the tenant is gone; replicas stop resolving it). Tenants created by migrations/seeds predate the producer and have no catalog history — the startup backfill (TenantCatalogBackfill, ordered after the authority + eventing migrations) emits them once, tombstones catalog keys whose row vanished behind the producer’s back, and is silent at steady state. The feed rides Authority itself: GET {authority}/catalog-changes/tenants under the same catalog:replicate machine scope.

Erasure semantics (GDPR, actor-identity). An erase is a versioned update, never a tombstone: the payload ships the already-nulled row with erased: true, so every replica converges to the erased state with zero PII on the wire. A tombstone would be wrong — a tombstoned replica row projects as unknown actor, and erased and unknown are DIFFERENT shapes in the three-shape identity projection. The SCR-4 offboarding cascade builds on this: erasure propagates to replicas through the ordinary replication path, no side channel.

5. Tenant lifecycle events (SCR-4, v1 producer half)

Typed offboarding events ride the SAME wire shape on their own stream, tenant_lifecycle, served by Authority’s feed under the same catalog:replicate scope. Consumers treat it as an ORDERED EVENT FEED (inbox dedup on (stream, epoch, seq)), never as versioned-row upserts.

Consumer contract on tenant.deleted: dispose the tenant’s data per your own P13 retention classes, with two hard overrides — evidence legal holds (evidence_holds) block disposal of held bundles, and ledger-class stores (Findings Merkle partitions) archive-then-detach rather than delete. Per DC-06, ignore unknown event types. The consumer-side handler surface + dry-run default land with the SCR-4 library half; per-service handlers ride each service’s program (or record an explicit not-applicable).

The evidence override, as actually implemented (011 EVD-11, 2026-08-18). EvidenceTenantLifecycleHandler (src/Evidence/StellaOps.Evidence.WebService/ TenantLifecycle/) is the named legal-holds adopter, and two measured facts refine the one-line rule above rather than contradicting it:

• “Block disposal of held bundles” is weaker than what the schema already guarantees. evidence_bundles_block_mutation raises EVIDENCE_BUNDLE_IMMUTABLE on EVERY delete of a sealed bundle, held or not, so no handler can dispose one in the first place. Of the 24 tenant-scoped tables in the consolidated database exactly TWO are derived-rebuildable (proofchain.graph_nodes, graph_edges); every other class is refused.
• The override therefore applies at TENANT granularity, not per bundle. No deletable table in the family carries a bundle reference, so “skip the held bundle’s rows” has no join to express it. An active hold — released_at IS NULL and no elapsed expires_at — suspends every deletion for that tenant, which is also what the baseline’s own P13 preamble asks for (“no class licenses deleting a held bundle or anything hanging off it”): the proof graph is what makes the held evidence interpretable.

A hold with a NULL bundle_id is unattributable and WIDENS the withholding rather than being ignored.

6. Consumer checklist (per SCR-3 flag flip)

1. Register AddCatalogReplication() (homes catalog_replica in your own DB) and AddEventingReliability() (inbox/checkpoints).
2. Wire HttpCatalogChangeFeed on a named client carrying your service credentials with the catalog:replicate grant.
3. Drain on your own schedule; read through CatalogReplicaReader, not CatalogReplicaStore directly — see §7 — and never through shared.* again.
4. The producer’s table stays the source of truth; your replica is rebuildable (drop + re-drain is always safe).

7. Reading a replica: three states, not two (SCR-3)

A replica read has three distinguishable outcomes, and collapsing the last two is the defect this section exists to prevent:

CatalogReplicaReader.LookupAsync returns that tri-state. SourceUnavailable covers a catalog_replica schema that has not converged, an unreadable/ unreachable database, and — the one a row count cannot see — a replica that has never drained. The witness for that last case is the drain’s durable checkpoint or its owner-confirmed empty observation, not a row count. A null stream epoch says only that the producer has not emitted; it does not prove that a producer’s domain table is empty before backfill. Without either durable witness, absence remains SourceUnavailable.

For a never-emitted stream, the producer may include emptyCatalogConfirmed:true only after its explicitly registered owner probe confirms an empty domain catalog. Platform opts in environment_state through its own IEnvironmentStateStore; other catalogs do not acquire this behavior automatically. The HTTP reader requires a matching catalog, explicit null streamEpoch and headSeq, an explicit empty changes array and the boolean confirmation. Missing/wrong-type/duplicate fields, contradictory confirmation, non-200 responses and failed authentication never establish an observation.

The consumer stores the confirmed observation in its own catalog_replica.empty_observations table (forward migration 002). It creates no event, epoch zero, seal document or Eventing checkpoint. The observation survives producer loss. A first real event removes it in the same local transaction as the row application and real checkpoint. Existing rows/checkpoints are preserved if a later response claims a never-emitted producer. Empty observations and event writes share a short local transaction advisory lock; no lock is held over HTTP. The consumer rechecks the checkpoint after acquiring that lock and discards a response whose starting checkpoint changed while the HTTP request was in flight. The reader obtains its row/checkpoint/empty-observation answer from one local repeatable-read snapshot, so a first-event race cannot combine an old missing row with a new checkpoint to invent absence.

empty_observations is derived-rebuildable: one row per observed empty catalog, removed on the first event or rebuilt by authenticated observation after an intentional replica reset. It is not a TTL cache, and it must not expire simply because the producer is offline. The migration adds no fixture rows or new runtime DDL writer; each existing AddCatalogReplication host owns its schema migration.

Reading by payload field (the reverse direction)

CatalogReplicaReader.LookupByPayloadFieldAsync answers the mirror question — which live row carries this value in its payload? — with the same three states, the same single repeatable-read snapshot and the same never-drained witness as LookupAsync. The tenants catalog needs it because the replica is keyed by the slug while the canonical UUID lives in the payload, so a consumer holding a UUID has no key to look up.

Two rules are specific to a reverse read:

• It is not unique by construction. PRIMARY KEY (catalog, key) says nothing about payload contents, so two live rows may carry the same value. That is SourceUnavailable, never a pick-the-first: a replica in that state cannot answer correctly, and choosing one would make an inconsistent replica look like a verdict.
• The field name is a bind parameter (payload->>@field), which is what makes it safe for a generic API. It also settles the index question: an expression index on (payload->>'id') can never be matched when the key is a parameter, so none is added. The existing partial index on (catalog) WHERE deleted = FALSE restricts the scan to one catalog’s live rows, which is what does the work.

The tenants-shaped accessor over it is IStellaOpsTenantKeyResolver (ReplicaStellaOpsTenantKeyResolver), registered by the same AddStellaOpsTenantResolverReplicaOnly call as the forward seam, so a flipped host acquires both directions with no extra configuration key. It is a separate interface on purpose: widening IStellaOpsTenantResolver would make every implementation owe a reverse it has no data for.

CatalogReplicaLookup.EnsureAnswered() converts SourceUnavailable into CatalogReplicaUnavailableException, whose message names the flag, the catalog:replicate grant and the drain. Callers whose return type cannot carry a third state — IStellaOpsTenantResolver returns Guid? — must throw rather than return the empty value, because a consumer reads that empty value as an answer: a null tenant surfaces as tenant_invalid_format (HTTP 400, blaming the caller) or “does not resolve to a known tenant” (HTTP 500 naming the wrong cause). That exact conflation cost the SPRINT_20260722_007 POL-F6 cutover two rollbacks: a gate evaluation decided correctly, was attested, and then failed its decision-history write with a message about the tenant identifier while the real fault was a missing Platform-owned relation.

Bootstrap window. Until the first drain commits a checkpoint or a confirmed empty observation, a missing-row read is SourceUnavailable — correctly, since the replica genuinely cannot answer. The tenants seam narrows that window by draining once, bounded, before host startup completes (Catalog:Replication:Tenants:BootstrapTimeout, default 30 s), ordered after the startup-migration hosts by registration order. It is deliberately non-fatal: a host whose replica has already converged must start even with a dead producer (the SCR-1 kill-the-producer posture).