ADR-040 - Zot Registry Retention and Boundary (keep the third-party lab/bundle registry; registry selector stays reserved)

• Status: Partially superseded by ADR-041 (owner ruling, 2026-08-03 — same day, later)
• Date: 2026-08-03

Superseding note (2026-08-03, ADR-041): the owner ruled that zot is demo/development-lab material only — NOT part of the release installation — and that the Stella Registry, Stella’s own product registry service, is built to back the stella+oci://registry selector and “take care of all” product-side registry roles including the meta-masking facade. Under that ruling: D1 (zot retained as third-party infrastructure) and D3 (registry stays reserved) are superseded; D4’s anonymous-path mechanics survive but their subject becomes the Stella Registry; §R1.3’s role statement is superseded (zot’s end state is demo integration, not Stella-artifact host — the stella-infra zot is only the interim plugin-publish target until the ADR-041 R6 cutover). D2’s boundary invariants, D5’s namespace/plan model, D6, and §R1.2’s destination classification survive and are carried forward by ADR-041. The inventory, measurements, and cost record below remain the authoritative point-in-time evidence base and are not rewritten.

• Owners: Lead architect / Integrations + ReleaseOrchestrator + Scanner + DevOps
• Supersedes: none
• Relates to: ADR-033, ADR-035, ADR-039; SPRINT_20260722_024 (registry token service fold); program ledger rounds 91-95; consolidation-design §4a/§4b

Context

The owner asked (2026-08-03, verbatim intent): “Can we delete zot? What is the zot infrastructure about? If it is for plugins and CLI, then we probably could create some anonymous repo path on the internal registry.” Ledger round 93 had already surfaced the companion architecture question: how — if at all — to provision the reserved stella+oci://registry backend selector (the internal primary image registry, the layer System of Record, which MaskedImageRefResolver fails closed on by design).

Three registry-shaped things exist today, and they are three different roles:

A full dependency sweep (2026-08-03, excluding frozen QA evidence under docs/qa/feature-checks/runs/** and archives) found 13 live consumers of zot — 7 push, 6 pull — plus the token-service auth plane:

Load-bearing measurements behind this decision:

1. Nothing in src/ production code resolves registry.stella-ops.local at runtime. Every source hit is help text, OpenAPI example: values, a UI wizard prefill, a lab-bootstrap manifest, or unit-test literals. The product features that touch zot are all registry-agnostic (config- or argument-driven).
2. The expensive capability is PUSH. Seven consumers use the distribution-spec push protocol (chunked/monolithic blob upload sessions, manifest PUT, tag lists) and three of them additionally depend on the OCI 1.1 Referrers API (build co-production verification, customer attach-referrers scripts, Scanner referrer dual-write).
3. The in-tree C# registry has no push path. OciRegistryEndpoints maps GET/HEAD /v2/ manifests+blobs only — verified by grep 2026-08-03: zero PUT/POST routes. It is a digest-verified pull-through/bundle cache, not a registry a docker push can target.
4. registry:2.8 (the second lab registry, docker-registry/oci-registry aliases, docker-compose.integrations.yml:243) is not a substitute: it lacks the Referrers API (GenericOciRegistryContainer.SupportsReferrersApi => false) and has no bearer auth, which would orphan the token service.
5. A namespace gap exists: the documented plugin publish target registry.stella-ops.local:5000/stellaops/plugins is covered by no plan in devops/etc/registry-token.yaml (stella-ops/public/* and stella-ops/enterprise/* only). Once bearer auth is enabled, plugin pushes and pulls 401 with no plan able to grant them.
6. The release bundle has drifted from the hardened posture: its registry-token env still carries a wildcard Plans__0__Repositories__0__Pattern: '*' with pull+push, the old api://registry audience shape, and Signing__KeyPath: /app/etc/certs/kestrel-dev.pfx — a dev TLS certificate doing duty as the token-signing key (devops/release/bundle/docker-compose.yml:3185-3192).

Decision

D1 - Zot stays, classified as third-party infrastructure

Refinement R1 (2026-08-03, same day — owner follow-up “zot theoretically is just stella ops and cli, nothing more”): the role list below was too generous. Role (i) “lab/dev registry for fixtures, e2e lanes, and CI” is not a legitimate zot role — the artifact-ownership classification in §R1 shows every fixture/e2e consumer stores a customer-application stand-in, which belongs on the external-registry simulator family, not on Stella’s own registry. Zot’s roles narrow to: (i) Stella’s own artifacts — plugin bundles and the plan-model distribution namespaces (stella-ops/public/*, stella-ops/enterprise/*); (ii) the evaluation-bundle registry, where it legitimately plays a dual role (Stella artifact store + the evaluator’s stand-in customer registry through the integrations path); (iii) the registry-token service’s subject. The keep verdict is unchanged; see §R1 for the evidence, the referrer-plane ruling, and the re-scoped rejected alternatives.

The zot registry is retained deliberately as a third-party infrastructure component of the same class as PostgreSQL, Valkey, SeaweedFS, and Rekor: something the installer provides so the platform has a real OCI distribution-spec registry (push protocol, Referrers API, GC, bearer auth) without Stella owning that conformance surface. Its roles are exactly three: (i) lab/dev registry for fixtures, e2e lanes, and CI (revoked by R1 — reassigned to the external-registry simulator family); (ii) the evaluation registry shipped in the release bundle; (iii) the plan-governed publish/pull target for plugin bundles in lab and evaluation estates. Zot is Apache-2.0 and pinned by digest in the bundle; no license-gate work is triggered.

D2 - Boundary invariants (what keeps zot deletable later)

• No production src/ code may dial registry.stella-ops.localor otherwise bind to zot at runtime. Today none does; conformance keeps it that way.
• Product OCI features remain registry-agnostic: registry targets arrive via configuration (Scanner host-alias pairs), required CLI arguments (--registry), or deployment config (pluginPull.registry) — never compiled-in hosts.
• Zot never becomes an identity anchor. Per ADR-033, masked stella+oci:// refs and digests are identity; zot hosts are routing coordinates that appear in no referrer subject key, audit identity, or release tuple.

Under these invariants, replacing or deleting zot is a compose + docs change, never a code change — which is the honest answer to “can we delete zot”: yes, mechanically, at any time; but nothing is gained today and real capability (push, referrers, the bundle evaluation on-ramp) would have to be rebuilt or abandoned.

D3 - stella+oci://registry remains RESERVED and is not backed by zot

The internal-primary selector stays fail-closed. Provisioning it is a product commitment — Stella becoming the layer System of Record — and no consumer in the inventory needs that today; the ADR-033 router re-introduction trigger (≥ 2 live backends × ≥ 2 consumers) is likewise unmet. Backing the reserved selector with the lab zot would silently promote lab infrastructure into product identity. Revisit only when a product requirement demands central layer custody (e.g. per-release masking pulls that agent pull-through cannot serve); the token-service plan model transfers unchanged to whatever backing is chosen then.

D4 - The “anonymous repo path” is a token-service plan, not new infrastructure

The capability the owner asked for already exists, shipped fail-closed: the token service can mint anonymous pull-only tokens (the Docker Hub/GHCR pattern) when anonymousPlan names a plan; the shipped config deliberately leaves it unset, pinned by ShippedRegistryTokenConfigTests. Concretely:

• Path shape: stella-ops/public/** is the only namespace an anonymous plan may ever cover (owner ruling B1, 2026-08-03). Anything meant to be anonymously pullable — public plugin bundles, CLI artifacts in an estate that mirrors them — is published under that prefix.
• Auth posture: anonymous callers get a pull-only token scoped to the anonymous plan; the service refuses to start if that plan grants anything but pull; all pushes remain authenticated everywhere; invalid credentials never downgrade to anonymous.
• Who publishes / consumes: operators and CI publish with plan-scoped tokens; deploy targets, CLI, and agents consume — anonymously only under stella-ops/public/** and only where an operator has opted the deployment in by setting anonymousPlan.
• Default: the flagship and customer masking registries stay token-only; the vendor’s public surface remains registry.stella-ops.org (already anonymous-pull).

Enabling the path is a config flip on an existing service. No new deployable, no new database, no new conformance surface — which is why deleting zot to get it would have been backwards: the anonymous path never required deleting anything.

D5 - In-flight token-service work is KEPT unchanged; the plugin namespace reconciles before bearer enablement

The B1/B1c work (zot bearer auth, the disabled anonymous capability, registry.token.issue scope catalog + seed, the smoke script, the gateway realm route) is kept as-is: zot is its subject, and the plan/token model is the proving ground for any future product registry (D3). One gap must close before the bearer-auth forcing function flips: the canonical plugin-bundle namespace moves under the single stella-ops/ root — stella-ops/plugins/<id>, token-gated via a dedicated publisher/consumer plan (plugin bundles are product content; signing + digest pinning provide integrity, plans provide distribution control). Plugins that become public distribution move under stella-ops/public/plugins/** and ride D4. Docs and CLI help examples follow. Executed via SPRINT_20260803_002.

D6 - The second lab registry is out of scope and stays

registry:2.8 (docker-registry/oci-registry.stella-ops.local, plus the per-provider htpasswd fixtures and CI mock registries) is test scaffolding for the Integrations connector surface and the auto-SBOM CI lane. It is retained as fixtures, is not part of the product boundary, and gains no auth plane. The stale docs that conflate its address with registry.stella-ops.local (which is zot) are corrected by the hygiene sprint.

R1 - Artifact-ownership refinement (2026-08-03): what zot stores vs what merely stands in

The owner’s model to test: masking customer registries is the internal registry’s job (stella+oci://registries/<slug> + agent pull-through), and zot’s legitimate content is Stella Ops’ own artifacts and the CLI — nothing more. The classification below confirms it: reclassified by what is actually stored, almost every fixture consumer is using zot as a stand-in for a customer’s registry.

R1.1 - The referrer-plane ruling (the sharpest question, settled by code)

Two readings were possible for Scanner’s SBOM/reachability referrer publishing: (i) the external attach targets wherever the subject image lives (zot = lab stand-in for the customer registry), or (ii) the meta plane should own referrers centrally and external publishing is a split-brain metadata plane. Both planes exist by design, and reading (i) is what the external leg means. There is no split-brain defect:

• The internal meta plane is the authoritative referrer home — exactly what ADR-033 D2 intends for SBOM/reachability/deployment-decision referrers. It is not a registry daemon: Deployment:Registry:MetaGatewayBaseUri points at the Scanner WebService, whose read-only /v2/{repo}/referrers|manifests|blobs/{digest} endpoints (OciReferrersReadEndpoints.cs) serve Postgres scanner.oci_referrers (schema comment: “system of record for SBOM/reachability referrers”) + the content-addressed object store. MetadataRegistryGateway is GET-only; writes arrive via the scan pipeline and POST /api/v1/oci-referrers (ReleaseOrchestrator deployment decisions ingest with publishExternally: false — “the internal store is the system of record”).
• The external leg is a gated projection onto the subject’s home registry. DualWriteOciReferrerCoordinator ALWAYS writes the internal row first (attached_internal), then — only when IRegistryPublishPolicy opts in — replays the byte-identical persisted manifest+blobs to the registry parsed from the scanned image’s own reference (SbomOciPublishStageExecutor.cs:194, ResolveRegistryHost(reference)), same repository per OCI referrers convention. Every gate defaults to false (RegistryHostAliases[n].PublishOciMetadata, the integration DTO’s publishOciMetadata, Integrations.Enabled), and the per-integration opt-in is the tell: in production the external target is the customer’s integrated registry, by the customer’s choice. Failure is loud (OciExternalPublishRequiredException fails the stage; retry job with backoff), and the internal copy survives regardless.

Ruling: the meta plane is the referrer home (reading (ii)'s intent, already true); the external attach is deliberate opt-in interop with the subject’s registry (reading (i)), which makes the lab’s 127.1.1.5 → zot alias-with-publish-on config zot playing a customer registry — the exact misuse the owner named. The fix is relocation of the lab target, not a code change. One adjacent gap for the record: Attestor’s OCI attach path targets external registries only and never writes the internal plane, so ADR-033’s “attestation referrers” listing for meta is aspirational today.

R1.2 - Per-consumer classification (consumer → stored → whose → verdict → destination)

R1.3 - Zot’s reduced role, and the claims that move with it

Once the misplaced consumers relocate, zot holds: plugin bundles + the plan-model distribution namespaces (stella-ops/public/*, stella-ops/enterprise/*) + the evaluation bundle’s dual role. The owner’s theory — “zot is just Stella Ops and the CLI” — is confirmed as the target state; today’s estate deviates only in the enumerated places, and every deviation is a config/default move, not a code change (except the DefaultRegistry compiled-in default).

Three context claims re-scope with this:

1. The Referrers API is no longer load-bearing on zot itself — the plugin path is manifest+blobs+digest only. Referrers become load-bearing on the simulator role (co-production proof, dual-write forcing, attach-referrers), so the rejection of registry:2.8 re-scopes: unfit as the simulator for referrer lanes; for the Stella-artifact store the blocker is bearer auth, not referrers. The cheapest referrers-capable simulator is another zot instance under a fixture alias (e.g. customer-registry.stella-ops.local, open/dev auth, own volume) — zot-the-image stays in the estate either way; what narrows is what THE Stella instance stores.
2. The 7-of-13-pushers argument recomputes but the verdict holds: after relocation zot’s own push surface is plugin publish + bundle evaluation pushes — still the real distribution-spec push protocol, still not something the pull-only agent registry or plain CAS can serve, and the simulator family still needs a push+referrers registry. Nothing about the reduction revives options (a) or ©.
3. The anonymous path is strengthened: reduced to Stella’s own artifacts, zot’s content is exactly the class eligible for stella-ops/public/** anonymous pull (D4). And relocation collapses the bearer-enablement credential matrix from thirteen consumers to roughly three (plugin publisher, plugin-pulling deploy agents, bundle evaluation flows) — fixture lanes move to the open simulator and need no zot credentials at all — which makes relocation the single best pre-flip simplifier for the SPRINT_20260722_024 S7 forcing function.

Consequences

• Ledger round 93’s open question is closed: zot is kept (option b), the reserved selector stays reserved, and no Stella-owned server-side registry is built.
• SPRINT_20260803_002_Registry_zot_boundary_hygiene.md carries the implied work: plugin-namespace/plan alignment (D5), release-bundle posture reconciliation (context item 6), doc-drift cleanup (D6 + LOCAL_SERVICES/README address conflation, module-doc pointer, testcontainer zot v2.1.1 vs compose v2.1.3 drift), and the bearer-enablement consumer matrix handed to SPRINT_20260722_024 S7 (every push/pull consumer named in the inventory gets a stated credential path before the flip — notably the Scanner worker referrer publish and the compliance-golden-path CI lane).
• R1 adds two tasks to the same sprint: RZB-5 (stand up the referrers-capable customer-registry simulator and relocate every misplaced consumer from §R1.2 — sequenced BEFORE the S7 flip, since it collapses the credential matrix) and RZB-6 (remove the compiled-in DefaultRegistry = "stellaops-registry:5000" / AllowInsecure = true defaults from Scanner worker options — the D2 violation §R1.2 found in product code).
• The evaluation bundle keeps its on-ramp story (“push a test image, scan it”) but converges to the same fail-closed token posture as the flagship stack.
• Anyone proposing a Stella-owned registry in future starts from this ADR’s cost record (below) and D3’s trigger, not from scratch.

Rejected alternatives

• (a) Delete zot; provision stella+oci://registry as a Stella-owned service by promoting StellaOps.Agent.Registry. Rejected on quantified cost: the agent code is pull-only (zero PUT/POST routes), so “promotion” means building the entire distribution-spec push surface (resumable chunked blob-upload sessions, cross-repo mounts, manifest PUT with validation), a referrers index, tag listing, deletion + GC, a storage backend, bearer-auth enforcement, and standing OCI conformance obligations — a new deployable microservice (ADR-039 P18 service-birth criteria unmet) re-implementing an Apache-2.0 CNCF single binary already digest-pinned in the bundle. Round 92 recorded the same rejection for the minimal in-house registry; this ADR extends it with the push-path evidence.
• © Delete zot without replacement. Rejected on the inventory: it removes the bundle’s evaluation on-ramp, breaks a live CI lane (compliance-golden-path) and the setup.sh bootstrap check (blocking-failure path), strands seven push fixtures and the documented plugin-release workflow, and leaves the just-hardened token service with zero subjects. The premise “zot is for plugins and CLI” did not survive contact with the inventory — plugins are one consumer of thirteen, and CLI distribution is already served anonymously by vendor infrastructure.
• Re-alias registry.stella-ops.local onto the existing registry:2.8. Rejected: loses the native Referrers API (silent degradation of referrer dual-write to the sha256-<hex> fallback-tag index) and bearer auth (orphans the token service and the B1 owner ruling).
• Back the reserved registry selector with zot now. Rejected per D3: promotes lab infrastructure into the product identity contract with no consumer needing it, and would make deleting zot later a product break instead of a compose change.
• Gateway-fronting the registry. Already rejected in round 92 (layer-blob throughput through the front door for zero authorization gain — the grants live in the token); re-recorded here so it is not re-litigated.