CI Docker Hub pull-through cache

Deploy and operate a self-hosted Docker Hub pull-through cache on the CI runner host so that container-image pulls (Testcontainers, docker pull, compose image references) are served from a local mirror — protected from upstream auth breakage and Docker Hub anonymous rate-limits, and pre-warmable for air-gap.

When to use this

Audience & prerequisites

0. STOPGAP — fix the broken runner Docker Hub creds (do this first)

The cache below is the durable fix, but the immediate unblock is to stop the runner Docker daemon from sending bad creds on anonymous public pulls. Bad creds in ~/.docker/config.json (or /root/.docker/config.json for the host-mode runner running as root) make Docker Hub answer 401 unauthorized even for public library/* images.

On the runner host, as the identity the daemon/runner uses (commonly root for the host-mode runner, the deployment user for the container runners):

# 1. Inspect what's there. The 'auths' map for docker.io / registry-1.docker.io
#    is the culprit if it carries a stale "auth" base64 cred.
cat ~/.docker/config.json
sudo cat /root/.docker/config.json   # host-mode runner runs as root

# 2a. Cleanest: log out the bad Docker Hub identity (removes only the docker.io entry).
docker logout
sudo docker logout

# 2b. Or surgically drop the docker.io auth entry while keeping other registries:
#     (requires jq; back up first)
cp ~/.docker/config.json ~/.docker/config.json.bak
jq 'del(.auths["https://index.docker.io/v1/"], .auths["registry-1.docker.io"], .auths["docker.io"])' \
  ~/.docker/config.json.bak > ~/.docker/config.json

# 3. Verify anonymous public pulls work again:
docker pull library/hello-world:latest && docker rmi library/hello-world:latest

If you intend to authenticate to Docker Hub (to raise the rate-limit) rather than go fully anonymous, replace the bad cred instead of removing it:

echo "$DOCKERHUB_TOKEN" | docker login -u "$DOCKERHUB_USER" --password-stdin

Once anonymous (or authenticated) pulls work, proceed to stand up the cache so the fix is durable and survives the next Docker Hub auth/rate-limit hiccup.

1. Deploy the pull-through cache

Infra-as-code lives in the repo:

On the runner host, from the repo checkout (/home/deployment/stella-ops.org/git.stella-ops.org):

cp devops/ci-local/.env.registry-cache.template devops/ci-local/.env.registry-cache

# (OPTIONAL) edit .env.registry-cache and set REGISTRY_PROXY_USERNAME +
# REGISTRY_PROXY_PASSWORD to a Docker Hub user + read-only PAT to lift the
# anonymous rate-limit. Leave blank for anonymous proxying.
$EDITOR devops/ci-local/.env.registry-cache

docker compose --env-file devops/ci-local/.env.registry-cache \
  -f devops/ci-local/docker-compose.registry-cache.yml up -d

Verify it is up and in proxy mode:

curl -fsS http://127.0.0.1:5000/v2/                 # -> {}
docker logs stellaops-dockerhub-cache 2>&1 | grep -i "proxy cache"
#   -> Registry configured as a proxy cache to https://registry-1.docker.io

# Warm one image end-to-end (the library/ infix is REQUIRED — see §4 note):
docker pull 127.0.0.1:5000/library/postgres:16-alpine
curl -fsS http://127.0.0.1:5000/v2/_catalog          # shows library/postgres

The cache stores blobs on the persistent named volume stellaops-dockerhub-cache-data, so it survives container recreation.

2. Wire it into CI — option (a): docker daemon registry-mirror (RECOMMENDED)

This is the transparent hook. The daemon rewrites every docker.io pull through the mirror, including the library/ namespace resolution for official images, so Testcontainers and docker pull inherit it with no test changes.

Edit /etc/docker/daemon.json on the runner host:

{
  "registry-mirrors": ["http://127.0.0.1:5000"]
}

If the cache is bound on a non-loopback interface or a different port, use that address. If you front the cache with TLS, use the https:// URL and ensure the daemon trusts the cert; an http:// mirror on loopback needs no extra config.

Apply:

sudo systemctl reload docker      # or: sudo systemctl restart docker
docker info | grep -A2 "Registry Mirrors"   # confirm the mirror is listed

After this, a plain docker pull postgres:16-alpine (no host prefix) is served by the cache automatically. This is the preferred wiring for the bare registry:2 proxy because the daemon handles the library/* resolution that the cache requires.

3. Wire it into CI — option (b): Testcontainers / explicit-pull prefix

For environments where you cannot change the daemon config, or where the mirror is a namespaced proxy that already maps library/* (e.g. Nexus :8083, or registry.stella-ops.local:5000/dockerhub), redirect image names instead.

Two repo-side knobs, set as Gitea org/repo variables (Settings → Actions → Variables), consumed by .gitea/workflows/test-manifest-execution.yml (full-sweep-host-infra job):

VariablePurpose
CI_DOCKERHUB_MIRROR_PREFIXUsed by .gitea/scripts/util/resolve-dockerhub-image.sh to rewrite the explicit docker pull of the Valkey image and any image exported via export-dockerhub-image-env.sh.
TESTCONTAINERS_HUB_IMAGE_NAME_PREFIXNative Testcontainers knob: prepends the prefix to every Docker Hub image the test suite pulls. No per-test code change.

Example values for a Nexus Docker Hub proxy:

CI_DOCKERHUB_MIRROR_PREFIX           = nexus.stella-ops.local:8083
TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX = nexus.stella-ops.local:8083/

Important library/ caveat (verified empirically). A bare registry:2 pull-through cache only serves official images under the library/ namespace: host:5000/library/postgres:16-alpine works, host:5000/postgres:16-alpine returns not found. Testcontainers’ HUB_IMAGE_NAME_PREFIX prepends the prefix to the short repository name (postgres → <prefix>postgres), which would miss the library/ infix. Therefore:

  • With a bare registry:2 cache → use the daemon mirror (§2). The HUB_IMAGE_NAME_PREFIX knob is best left unset for that cache.
  • With a namespaced proxy (Nexus, or a registry namespace) that maps library/* for you → HUB_IMAGE_NAME_PREFIX works directly.

The trailing slash matters for the Testcontainers prefix: it is concatenated verbatim before the image repository.

Runner job-image bootstrap (nested container runners)

The ubuntu-latest container runners spawn a nested node:20-bullseye job container. Seed it from the mirror so job startup never cold-pulls Docker Hub:

CI_DOCKERHUB_MIRROR_PREFIX=nexus.stella-ops.local:8083 \
  bash .gitea/scripts/util/seed-runner-job-image-cache.sh

See .gitea/docs/runners.md for the act_runner label mapping if container.force_pull is enabled.

4. Pre-warming / air-gap seeding (offline-first)

The cache is a normal registry:2 store, so it can be pre-warmed before an air-gap window and will keep serving cached images with the upstream unreachable.

Pre-warm online (run while the host still has Docker Hub egress). With the daemon mirror (§2) configured, simply pull the images CI needs; they land in the cache volume:

for img in \
  postgres:16-alpine \
  valkey/valkey:7-alpine \
  registry:2.8 \
  ; do docker pull "$img"; done

Without the daemon mirror, address the cache directly (mind the library/ infix for official images):

docker pull 127.0.0.1:5000/library/postgres:16-alpine
docker pull 127.0.0.1:5000/valkey/valkey:7-alpine

Seed from an offline bundle. The cache volume (stellaops-dockerhub-cache-data) is portable. To move a warmed cache into an air-gapped host, tar the volume on a connected host and restore it on the target:

# On the connected host, after warming:
docker run --rm -v stellaops-dockerhub-cache-data:/data -v "$PWD":/out alpine \
  tar czf /out/dockerhub-cache-seed.tgz -C /data .

# Transfer dockerhub-cache-seed.tgz across the air-gap, then on the target host
# (with the cache stopped):
docker run --rm -v stellaops-dockerhub-cache-data:/data -v "$PWD":/in alpine \
  tar xzf /in/dockerhub-cache-seed.tgz -C /data
docker compose --env-file devops/ci-local/.env.registry-cache \
  -f devops/ci-local/docker-compose.registry-cache.yml up -d

This complements the broader offline image mirror tooling under devops/offline/airgap/ (build_mirror_bundle.py), which packages and signs mirror bundles for the full air-gap kit. The pull-through cache is the serving side; those tools are the bundle side.

5. Operate & maintain

Failure modes

SymptomCauseFix
unauthorized: incorrect username or password on a public pullBad creds in runner ~/.docker/config.json§0 stopgap — docker logout / drop the docker.io auth entry
host:5000/postgres:16-alpine: not found but library/postgres worksregistry:2 proxy requires the library/ infix for official imagesUse the daemon mirror (§2) so the daemon adds library/, or address library/<image> explicitly
Cache serves stale tag after upstream updateregistry:2.8 keeps cached manifests (no env TTL)Recreate the cache volume, or mount a config.yml with proxy: { ttl: ... }
REGISTRY_PROXY_TTL warning in logsregistry 2.8.x ignores that env varHarmless; remove it or use a mounted config.yml for a finite TTL
Daemon mirror ignoreddaemon.json not reloaded, or TLS mismatchsudo systemctl reload docker; verify with docker info