Registry bridge pull-through cache

Operate the Stella Ops agent registry bridge — the loopback OCI registry an agent exposes so deploy pulls for masked stella+oci://registries/<slug>/… references are served from a local pull-through cache instead of hitting the real upstream every time. This runbook covers enabling the bridge, configuring upstreams — agent-side static config and the central control-plane CRUD (API / CLI / UI) — and cache policy, the L1/L2 tiers, eviction and pinning, the opt-in central upstream-map and policy refresh, and hit/miss/eviction troubleshooting.

When to use this

• You are standing up an agent that deploys images by masked reference and want layer pulls cached locally (faster rollouts, fewer upstream round-trips, resilience to upstream throttling).
• You run an air-gapped (BundleOnly) agent whose only image source is an imported OCI bundle, and need the cache to never evict that content.
• You are tuning the cache size cap / eviction watermarks, or diagnosing a low hit ratio, eviction churn, or upstream-fetch failures on the bridge cache dashboard.

Audience & prerequisites

• Operator with access to the agent host and its configuration (Agent:LocalRegistry:* — appsettings / environment / sealed deploy bundle).
• For cache-policy authoring: a token carrying registry:cache:admin (read views need registry:read). ADMIN implies both.
• For central upstream (bridge) authoring: a token carrying registry.admin (note the dot, not a colon — distinct from registry:cache:admin); read views need registry:read. The scope catalog is src/Authority/StellaOps.Authority/StellaOps.Auth.Abstractions/StellaOpsScopes.cs (RegistryRead = registry:read, RegistryCacheAdmin = registry:cache:admin, RegistryAdmin = registry.admin).
• For metrics: the agent’s meter StellaOps.Agent.Registry exported to Prometheus, and the Grafana board imported (see §7).

1. Enable the agent registry bridge

The bridge is configured under Agent:LocalRegistry (binding type StellaOps.Agent.Registry.LocalRegistryOptions, wired by AddLocalOciRegistry(...) in the agent host). It listens loopback-only — a non-loopback BindAddress is rejected at startup (fail-closed) so the cache is never exposed off-host.

Minimal pull-through config (appsettings or env):

{
  "Agent": {
    "LocalRegistry": {
      "Enabled": true,
      "Mode": "PullThrough",          // or "BundleOnly" for air-gap
      "BindAddress": "127.0.0.1",     // loopback only; ::1 also allowed
      "Port": 8021,
      "CacheRoot": "/var/lib/stellaops/agent-registry-cache"
    }
  }
}

Environment-variable form (double-underscore section separator):

export Agent__LocalRegistry__Enabled=true
export Agent__LocalRegistry__Mode=PullThrough
export Agent__LocalRegistry__Port=8021
export Agent__LocalRegistry__CacheRoot=/var/lib/stellaops/agent-registry-cache

Set a durable CacheRoot. If unset, the cache root defaults to <LocalApplicationData>/StellaOps/stellaops-agent-registry-cache — deliberately not %TEMP%, because host maintenance wipes temp and that would silently cold-start the cache (and, in BundleOnly mode, break an air-gap agent whose only image source is the imported bundle). Point CacheRoot at a persistent host path you back up with the rest of the agent state.

On startup the agent rebuilds its in-memory cache index from whatever content is already on disk under CacheRoot (survives restarts; a rebuild failure degrades to an empty index rather than crashing boot).

If Enabled is false and no OciLayoutBundlePath is set, the bridge is inert (a no-op hosted service) — safe default.

2. Configure upstreams (which real registry a slug bridges to)

A masked pull for registries/<slug>/… resolves to a configured upstream host; an unknown (or Enabled:false) slug is refused (fail-closed). The agent’s resolver (ConfigUpstreamRegistryResolver) reads the slug → host map; that map is provisioned from agent-side static config under Agent:LocalRegistry:Upstreams and may optionally be refreshed from the control plane (§2.3).

“No live add-upstream API” — true for the agent, not for the orchestrator. There is intentionally no mutation surface on the agent loopback — the agent never exposes an “add upstream” endpoint, so an air-gap agent’s routing table stays immutable and auditable. What does exist is a central control-plane CRUD (/api/v1/registry/upstreams, §2.2): operators author the bridge map centrally; connected agents opt in to pull it (§2.3); air-gap agents ignore it and keep their static Agent:LocalRegistry:Upstreams.

2.1 Agent-side static config (Agent:LocalRegistry:Upstreams)

{
  "Agent": {
    "LocalRegistry": {
      "Upstreams": [
        {
          "Slug": "primary-gitlab",            // the masked-ref slug
          "Host": "registry.example.com",      // real upstream host
          "Enabled": true,
          "RepositoryPrefix": "example-corp/app" // optional path prefix
        }
      ]
    }
  }
}

A pull for stella+oci://registries/primary-gitlab/core:dev is then served by the bridge, fetched (on a miss) from registry.example.com under the optional prefix. This is the only routing source an air-gap agent uses: sealed with the agent, immutable at runtime, no network needed.

2.2 Central control-plane CRUD (/api/v1/registry/upstreams)

Operators author the slug → host bridge map centrally in the orchestrator (RegistryUpstreamEndpoints); the same UpstreamRegistry model the agent reads. A bridge is install-wide (no tenant scoping — the read view is not tenant-scoped). Authoring is fail-closed: see the slug and credentialRef rules below.

API (base path /api/v1/registry/upstreams):

UpstreamDto (camelCase over the wire): { slug, host, enabled, repositoryPrefix?, credentialRef? }.

Slug-format rule (fail-closed). slug must match ^[a-z0-9][a-z0-9-]*$ (lowercase letters, digits, and hyphens; no /, dots, or whitespace). The slug is the bridge key in registries/<slug>/<objectPath>, which the agent splits on the first /to separate the slug from the upstream object path — a / (or whitespace) in the slug would corrupt that split, so it is rejected at PUT.

credentialRef must be a secret reference (fail-closed). When supplied, credentialRef must start with a recognised secret-reference scheme — builtin://, vault://, openbao://, authref://, secret://, file://, base64:, or plain:. A bare user:password (a colon but no recognised scheme) is rejected, so raw credentials never land in the bridge table; the agent resolves the reference at pull time via the ADR-034 secret-access broker. The credentialRef is also never returned on GET /effective— upstream creds stay broker-resolved and never leave the control plane.

On-prem-only host rule (fail-closed). host must be a self-hosted / on-prem registry. Public cloud-managed registries (*.amazonaws.com, *.azurecr.io, gcr.io, *.pkg.dev) are rejected with a clear 400 (AGENTS.md §1.5 on-prem posture; host:port is checked by host only).

CLI (stella registry upstream)

# Read the central bridge map
stella registry upstream list
stella registry upstream list --json

# Create / update a bridge (credential-ref is a secret reference, never raw creds)
stella registry upstream set \
  --slug primary-gitlab \
  --host registry.example.com \
  --prefix example-corp/app \
  --enabled true \
  --credential-ref vault://registry/primary-gitlab

# Delete a bridge by its slug
stella registry upstream delete --slug primary-gitlab

--enabled defaults to false when omitted (a freshly-set bridge is disabled — and disabled slugs are fail-closed on pull — until you enable it). The CLI rejects a non-secret --credential-ref client-side (before the request is sent) with the same scheme list as the backend, so a raw user:password never leaves the operator’s shell.

The integration side (UI / CLI)

Upstream registry integrations (the real registries Stella discovers/scans images from, and the slugs the masked refs reference) are managed in the Integrations → Registry Admin console, and inspected from the CLI:

# List registry integrations and discovered-image posture
stella registry list
stella registry list --provider example-corp --json

# Resolve a masked ⇄ real reference (no backend call)
stella registry resolve "stella+oci://registries/primary-gitlab/core:dev"
stella registry resolve "registry.example.com/example-corp/app/core:dev" --reverse

Console route: /ops/integrations/registry-admin (and /setup/integrations/registry-admin); the Upstreams tab is the UI for the central CRUD above (read needs registry:read; the add/edit/delete controls are shown only with registry.admin). Keep the slug used in Upstreams consistent with the integration slug so masked refs resolve end-to-end.

2.3 Opt-in agent pull of the central map (UpstreamRefresh)

A connected agent can periodically pull the orchestrator-resolved effective upstream map (GET /api/v1/registry/upstreams/effective) and apply it live to its running resolver — the agent-side analogue of the cache PolicyRefresh (§6). Off by default (air-gap-first): when disabled (or for an air-gap agent), the agent keeps its provisioned Agent:LocalRegistry:Upstreams and never reaches the network for routing.

{
  "Agent": {
    "LocalRegistry": {
      "UpstreamRefresh": {
        "Enabled": true,
        "OrchestratorBaseUrl": "https://orchestrator.internal",
        "IntervalSeconds": 300         // floored at 30 by the service
      }
    }
  }
}

• The effective map is the non-secret slug → host view (slug, host, enabled, repositoryPrefix) — credentialRef is never on the wire; creds stay broker-resolved at pull time.
• Fail-soft + fail-closed: any non-success / parse / transport error keeps the current map (an unreachable orchestrator never drops the agent’s bridges nor points one at the wrong host); a 401/403 is logged distinctly as a likely missing/wrong agent registry:read credential. The swap is atomic — a concurrent pull never observes a half-built map — and a slug dropped from the new map resolves as unknown (fail-closed), never a silent fallback.

3. Configure cache policy (size cap, watermarks, idle, enable)

Cache policy (size budget + eviction behaviour) is authored centrally in the orchestrator and is scoped: Global < Tenant < Upstream < Repository, with more specific scopes overriding individual fields (field-level fallthrough). The resolved effective policy is what an agent applies.

API (base path /api/v1/registry/cache):

PUT validates fail-closed before persisting: scope-key fields are required for their scope (e.g. TenantId for Tenant+, UpstreamSlug for Upstream+, Repository for Repository), MaxCacheBytes >= 0 (0 = unbounded), watermarks in 1..100, and LowWatermarkPercent < HighWatermarkPercent.

CLI (stella registry cache)

# Read
stella registry cache list-policies
stella registry cache list-policies --json

# Author a Global 20 GiB cap with default 90/75 watermarks
stella registry cache set-policy --scope Global \
  --max-bytes 21474836480 --high 90 --low 75 --enabled true

# Tighter cap for one upstream slug under a tenant
stella registry cache set-policy --scope Upstream \
  --tenant acme --upstream primary-gitlab --max-bytes 5368709120

# Delete a scoped policy by its key
stella registry cache delete-policy --scope Upstream \
  --tenant acme --upstream primary-gitlab

--max-idle-seconds is accepted and stored on the policy (it flows through the contract, CLI, UI, and the resolved effective policy). Note: the agent’s live L1 evictor enforces the size cap + watermarks only; idle-based eviction is a policy tunable that the loopback LRU manager does not yet act on. Size your MaxCacheBytes accordingly — do not rely on MaxIdle to bound the cache today.

UI

Integrations → Registry Admin → Cache (registry-admin/cache). The read view needs registry:read; the New Policy / edit / delete controls are shown only with registry:cache:admin (UI gating is convenience — the endpoint enforces the scope regardless).

How an agent gets the policy

• Air-gap / provisioned: the agent uses its local Agent:LocalRegistry:Cache config (MaxCacheBytes, HighWatermarkPercent, LowWatermarkPercent), or the effective policy sealed into the deploy bundle. No network needed.
• Connected (opt-in): see §6 PolicyRefresh — the agent periodically pulls the resolved effective policy and applies it live.

Defaults (built-in floor, matching the agent’s CacheOptions defaults): Enabled=true, MaxCacheBytes=0 (unbounded), HighWatermarkPercent=90, LowWatermarkPercent=75.

Unbounded + durable root = unbounded disk. With MaxCacheBytes=0 the cache never evicts and grows with every unique layer. Set a real cap on any host where disk is finite. Recommended starting point: a few times your largest deploy’s total layer size.

4. L1 / L2 tiers

• L1 — local filesystem (always present when the bridge is enabled). The durable CacheRoot store, wrapped by the eviction-managing RegistryCacheManager. This is the source of truth for residency on the agent and the tier eviction acts on.
• L2 — optional shared/central store. When configured, an L1 miss reads from L2 and, on an L2 hit, promotes the content into L1 (so subsequent reads are local and tracked by the index/evictor). Writes are best-effort write-through to L2: an L2 failure is logged but never fatal — L1 keeps the content and the deploy proceeds. An L2 read failure degrades to a miss (the caller pull-throughs upstream).

Read path: L1 → L2 (promote on hit) → upstream pull-through. Each tier that serves the content records a hit; only the final upstream fetch increments upstream_fetch.

5. Eviction & pinning

Size-capped LRU with hysteresis. When MaxCacheBytes > 0, after a put that pushes usage over the high watermark, eviction runs and drains the cache down to the low watermark (the hysteresis band avoids evict-on-every-put churn). Eviction selects least-recently-used victims; GET bumps last-access so hot content survives. Eviction is serialized so concurrent loopback puts cannot double-delete. MaxCacheBytes=0 means unbounded — eviction is disabled.

Pinning (never evicted):

• Bundle-imported (air-gap) content is pinned intrinsically. When OciLayoutBundlePath is set, the agent verifies (sha256) and imports the OCI layout, and that content is pinned — eviction skips it regardless of the cap. An empty/aggressive policy can never evict an air-gap agent’s only image source.
• Operator pins (RegistryCacheManager.Pin(digest)) mark a digest so eviction never removes it.

Applying a smaller cap drains immediately. When a new effective policy (via PolicyRefresh, §6) shrinks MaxCacheBytes, the agent recomputes the watermarks and runs eviction right away, draining down to the new low watermark.

6. Opt-in central policy refresh (PolicyRefresh)

Connected agents can pull the orchestrator-resolved effective policy on an interval and apply it live. Off by default (air-gap-first): when disabled, the agent keeps its provisioned Agent:LocalRegistry:Cache.

{
  "Agent": {
    "LocalRegistry": {
      "PolicyRefresh": {
        "Enabled": true,
        "OrchestratorBaseUrl": "https://orchestrator.internal",
        "IntervalSeconds": 300,     // floored at 30 by the service
        "TenantId": "acme",          // scope keys used to resolve the policy
        "UpstreamSlug": "primary-gitlab",
        "Repository": "example-corp/app/core"
      }
    }
  }
}

The agent GETs /api/v1/registry/cache/policies/effective?tenantId=…&upstreamSlug=…&repository=… and applies MaxCacheBytes / HighWatermarkPercent / LowWatermarkPercent to the running cache. Fail-soft: any non-success, parse, or transport error keeps the current policy — an unreachable orchestrator never makes the cache go unbounded or wrong. (MaxIdleSeconds is returned but not applied; see §3.)

This is a pull channel only — there is no live push. Air-gap agents do not enable it; they rely on provisioned config or the sealed bundle.

7. Observability dashboard

Import devops/telemetry/dashboards/stella-ops-registry-bridge-cache.json (uid stella-ops-registry-bridge-cache) into Grafana — same Prometheus ${datasource} convention as the other boards under devops/telemetry/dashboards/.

Metrics come from the meter StellaOps.Agent.Registry (RegistryCacheTelemetry). OTel dotted names export to Prometheus underscored with a _total suffix on counters:

Panels: hit ratio (gauge + over-time by kind), hit-vs-miss request rate, evictions over time, single-flight coalesces over time, upstream-fetch outcome (ok/error) and upstream error ratio.

Key PromQL (matches the dashboard):

# Hit ratio
sum(rate(stella_registry_cache_requests_total{result="hit"}[$__rate_interval]))
  / clamp_min(sum(rate(stella_registry_cache_requests_total[$__rate_interval])), 1e-9)

# Upstream fetch error ratio
sum(rate(stella_registry_cache_upstream_fetch_total{outcome="error"}[$__rate_interval]))
  / clamp_min(sum(rate(stella_registry_cache_upstream_fetch_total[$__rate_interval])), 1e-9)

Telemetry is optional — when the meter is absent the cache records nothing. Confirm StellaOps.Agent.Registry is in the agent’s OTel meter allowlist and scraped by Prometheus before relying on these panels.

8. Troubleshooting (hit / miss / eviction)

9. Digest-centric content broker (agent holds no L2 key, no upstream credential)

§1–§8 describe a standalone pull-through cache where the agent itself fetches upstream (it resolves a credentialRef via the ADR-034 secret broker at pull time). The agent-as-registry deploy path is stricter: on the deploy path the edge agent holds no standing upstream credential and no L2 key at all — it obtains image content by digest from the orchestrator over its authorized mTLS channel, and the orchestrator alone holds the per-upstream credentials and the L2 store key. This section documents that digest-centric model (see ADR-035).

9.1 Why digest is the unit of authorization

Releases are digest-pinned — the release pins the image manifest digest (ADR-033 §D1: digest is identity, masking is routing). So a content digest is just another allowed reference: the minted deployment capability (DeploymentSecretCapability, see ADR-034 §D1) already authorizes a set of reference-hashes via AllowedRefHashes / AllowsRef(rawRef) / AllowsRefHash(hash) / ComputeRefHash(raw). Scoping the capability on the pinned digest needs no schema change — the digest is added as an allowed reference at mint time (DeploymentSecretModeResolver.TryMintPullCapabilityAsync / DeploymentVarsBundleAssembler.ComputePullModeSecretReferencesAsync). The capability is sealed under the master KEK (Seal/Open) so the agent can neither forge nor widen it.

Because both the authorization and the cache key are the digest, an agent can only ever pull content it is named for, and identical content dedups to one object regardless of which upstream/agent produced it.

9.2 Who holds what

The agent never opens the seal — it relays it back to the orchestrator, which opens + validates it (Open + AllowsRef(digest)). This shrinks the agent’s blast radius to “this deployment’s pinned digest, until this deployment ends or the short TTL elapses” — consistent with the ADR-034 fail-closed posture.

9.3 Read path (content-by-digest)

On a pull-through miss the agent fetches by digest from the orchestrator:

GET /api/v1/release-orchestrator/deployments/{deploymentId}/registry/content/{kind}/{digest}
    {kind} = manifest | blob          {digest} = sha256:<64 lowercase hex>
    X-StellaOps-TenantId: <tenant GUID>
    X-StellaOps-Deployment-Capability: <opaque sealed capability>
    (over the agent's mTLS channel)

The orchestrator processes the request fail-closed, mirroring the mTLS-only, tenant-checked, cross-tenant-404 DeploymentSecretBrokerEndpoints precedent:

1. Open + validate the capability with Stella crypto (AEAD tag, tenant + deployment AAD binding, ExpiresAt); active-check the deployment (terminal ⇒ reject — this is revocation).
2. Scope-check the requested digest against AllowsRefHash(ComputeRefHash(digest)). Out of scope ⇒ 403(fail-closed); capability absent/invalid ⇒ 401.
3. Serve from L2: read the content-addressed object from the RustFs L2 store (RustFsObjectStoreClient → ObjectStoreOciContentStore, key = the digest).
4. Fetch-on-miss: if not in L2, resolve the private upstream credential (orchestrator-held, via the routing ISecretProvider), fetch by digest, digest-verify the bytes, write-through to L2, and return. 404 only when the content is in neither L2 nor any reachable upstream.

The bytes are digest-verified by the orchestrator before serving and re-verified by the agent on receipt (defence-in-depth) — the digest is the only integrity anchor the edge agent has (it holds no upstream trust), so unverified bytes are refused fail-closed and never stored. A Docker-Content-Digest header mismatch is treated as a server contract violation (fail-closed).

9.4 The cache is digest-keyed (content-addressed, global dedup)

This is already true and is the property the broker relies on:

• L1 (the agent’s filesystem CacheRoot, §4): the index + eviction operate on digests; GET/PUT are keyed on sha256:<hex>.
• L2 (the orchestrator’s RustFs store, §4): ObjectStoreOciContentStore derives the object key purely from the digest (<prefix>/{manifests|blobs}/sha256/<hex>), so the same content dedups to one object fleet-wide regardless of upstream/agent. The pull-through path refuses non-digest references — there is no tag-keyed entry to poison.

Because the broker token is scoped on digest and the cache is keyed on digest, authorization and storage share one coordinate end-to-end: the agent can request exactly (and only) the digests its capability names, and every such request hits a content-addressed object.

9.5 Orchestrator-side L2 config (RustFs endpoint + key as a secret-ref)

The orchestrator-side L2 store is configured on the release-orchestrator service (devops/compose/docker-compose.stella-services.yml); the env-substitution defaults live in devops/compose/env/stellaops.env.example. The L2 key is a secret reference (ADR-032), never a raw value — it resolves through the same RoutingSecretProvider the service already wires (Crypto__SecretProviders__*).

Never put the L2 key value in compose or env. Only a secret reference belongs in config; the orchestrator resolves it at runtime. A raw value in ApiKeyRef is a contract break (and a supply-chain leak). The recognised schemes are builtin://, vault://, openbao://, authref://, secret://, file://, base64:, plain:.

Related

• docs/architecture/decisions/ADR-035-digest-centric-content-broker.md — the digest-centric content broker (no standing edge secret; content-addressed authorization; agent fetches by digest over mTLS). Read alongside §9.
• docs/architecture/decisions/ADR-033-unified-registry-access.md — masked stella+oci:// references and the unified registry-access model.
• docs/architecture/decisions/ADR-034-agent-secret-access-broker.md — how the agent resolves an upstream credentialRef at pull time (the secret never lands in the bridge table or on the effective map).
• docs/runbooks/ci-registry-pull-through-cache.md — the CI Docker Hub pull-through cache (different concern: CI runner image pulls, not the agent bridge).
• docs/runbooks/registry-compatibility.md — OCI registry push/pull compatibility checks.
• devops/telemetry/dashboards/stella-ops-registry-bridge-cache.json — the Grafana board this runbook references.
• Code: src/ReleaseOrchestrator/__Agents/StellaOps.Agent.Registry/ (bridge + cache; UpstreamRegistryResolver.cs slug → host resolver, UpstreamRefreshHostedService.cs opt-in §2.3 pull), src/ReleaseOrchestrator/__Apps/StellaOps.ReleaseOrchestrator.WebApi/Endpoints/RegistryCacheEndpoints.cs (policy API), src/ReleaseOrchestrator/__Apps/StellaOps.ReleaseOrchestrator.WebApi/Endpoints/RegistryUpstreamEndpoints.cs (central upstreams CRUD, §2.2), src/Cli/StellaOps.Cli/Commands/RegistryCommandGroup.cs (stella registry / stella registry cache / stella registry upstream), src/Web/StellaOps.Web/src/app/features/registry-admin/components/cache-policy.component.ts and upstream.component.ts (UI).
• Code (digest-centric broker, §9): src/ReleaseOrchestrator/__Agents/StellaOps.Agent.Core/Registry/OrchestratorContentByDigest.cs and HttpOrchestratorContentByDigestClient.cs (agent fetch-by-digest over mTLS), src/ReleaseOrchestrator/__Libraries/StellaOps.ReleaseOrchestrator.RegistryObjectStore/RustFsObjectStoreClient.cs and src/ReleaseOrchestrator/__Agents/StellaOps.Agent.Registry/Cache/ObjectStoreOciContentStore.cs (digest-keyed L2), src/ReleaseOrchestrator/__Libraries/StellaOps.ReleaseOrchestrator.Deployment/Secrets/DeploymentSecretCapability.cs (the capability scoped on the digest).