Publishing the evaluation bundle — step by step

How to cut registry.stella-ops.org/stellaops/* images and the get.stella-ops.org download for a release of the evaluation bundle, from a clean tree to a published, verifiable artefact set.

Written for someone doing this for the first time. Every step says what it does, how to tell it worked, and what to do when it did not.

Scope: the customer-facing evaluation bundle in devops/release/bundle/. For the production-rollout guardrails a reviewer signs off before a tag ships, see release-checklist.md — that is a different document and both apply.

Current RC1 operator card

Use this path for a normal repeat release. The detailed sections below explain a gate or recovery only; they are not a second workflow.

1. Fix the source before touching production. Product and website changes must be committed on clean, pushed main branches. Record both SHAs. A temporary branch may carry review work, but production is never built from it.

2. Preflight every machine that verifies or signs. command -v cosign and cosign version must pass on the build host and the clean acceptance host. Confirm that the signing key’s public half matches https://stella-ops.org/keys/cosign.pub.

3. Build, publish, and sign once. Prepare the release plugin inputs (§2), then run the single build-and-publish.sh --push --sign command in §4. Stop unless the bundle verifier reports every file and every product image digest.

4. Publish immutably. Package with §5, stage beside the download docroot, verify the remote checksums, then move the version and latest/ trees into place (§6). Never delete an older version and never rsync --delete the release history.

5. Publish the website from pushed main. Let the existing website Build Test Deploy workflow deploy that SHA to staging, accept it, then run Promote Site (Manual). That promotion updates both www.stella-ops.org and the apex site. Do not upload a second locally built website artifact. If the private Actions UI is unavailable, use the existing deployment SSH account: build once from pushed main, write parseable metadata containing that SHA and refs/heads/main, sync it to staging, validate it there, then server-side rsync that exact staged tree to both production docroots. Do not rebuild between environments.

6. Run the public procedure from a new directory. The acceptance host must have Cosign on PATH before starting:

   curl -fsSLO https://get.stella-ops.org/releases/$STELLA_VERSION/stellaops-bundle-$STELLA_VERSION.tar.gz
   curl -fsSL https://get.stella-ops.org/releases/$STELLA_VERSION/SHA256SUMS \
     | sha256sum --check --ignore-missing
   tar xzf stellaops-bundle-$STELLA_VERSION.tar.gz
   cd stellaops-bundle-$STELLA_VERSION
   curl -fsSL https://stella-ops.org/keys/cosign.pub -o public-release-signing.pub
   cmp public-release-signing.pub release-signing.pub
   cosign verify-blob --insecure-ignore-tlog=true --key public-release-signing.pub \
     --signature release-manifest.yaml.sig release-manifest.yaml
   python tools/verify-bundle.py --require-signature --require-digests
   ./install.sh --no-start
   docker compose --env-file .env -f docker-compose.pinned.yml \
     -f secrets/docker-compose.installation.yml config --quiet
   

   The explicit tlog flag is required because this release uses a pinned offline key without a Rekor entry. It skips the online log lookup; Cosign still verifies the detached signature against the independently fetched public key. Run one clean full ./install.sh acceptance before announcing a newly built bundle.

7. Close only after source and live state agree. Check live deployment metadata, apex and www, every localized /install/ route, the public key, and the download. Archive the sprint and delete the temporary branch only when its commits are reachable from pushed main and production reports that main SHA.

This card intentionally adds no infrastructure. It uses the existing registry, download vhost, website staging workflow, and manual production promotion.

0. Before you start

You need, and no agent can supply any of these:

Working tree: clean, on the commit you intend to release. The build stamps the git SHA into every image (/buildinfo.json), so a dirty tree produces images whose provenance points at a commit that does not exist anywhere else.

Decide the version. It appears in image tags, the manifest, the bundle filename and the download URL. Semver-ish, never latest — the tooling refuses latest explicitly, but it does not validate the shape of anything else, so a typo here publishes silently under the wrong tag.

The canonical release string is v1.0.0-RC1— the same value migration 092_PlatformVersionSetting.sql seeds into platform.environment_settings and the console sidebar renders. Use it exactly; 1.0-RC1 and 1.0.0-RC1 are different tags and will not match the version the product reports about itself.

export STELLA_VERSION=v1.0.0-RC1
export STELLA_BUNDLE_CHANNEL=rc

Two forms, deliberately. v1.0.0-RC1 is the release/tag/display form: git tag, image tag, bundle filename, download URL, console sidebar. The bare 1.0.0-RC1 is the assembly/package form used in .csproj metadata and NuGet, because MSBuild rejects a leading v (error : 'v1.0.0-RC1' is not a valid version string). Both come from src/Directory.Versions.props, which defines StellaOpsReleaseVersion (bare) and StellaOpsReleaseTag (v-prefixed). Do not “normalise” one into the other.

1. Regenerate the bundle from the dev stack

The compose files are generated, never hand-edited. Regenerate first so the bundle matches the stack as it is now.

cd devops/release/bundle
python tools/generate-bundle.py       --version "$STELLA_VERSION" --channel "$STELLA_BUNDLE_CHANNEL"
python tools/generate-bundle.py       --pin-template --version "$STELLA_VERSION" --channel "$STELLA_BUNDLE_CHANNEL"
python tools/stage-config.py
python tools/stage-plugin-layers.py   plan --version "$STELLA_VERSION" --channel "$STELLA_BUNDLE_CHANNEL"

Verify:

python tools/generate-bundle.py     --check --version "$STELLA_VERSION" --channel "$STELLA_BUNDLE_CHANNEL"
python tools/stage-plugin-layers.py check  --version "$STELLA_VERSION" --channel "$STELLA_BUNDLE_CHANNEL"
python tools/stage-config.py        --check
docker compose -f docker-compose.yml --env-file .env.example config >/dev/null

All four must succeed. The last one is the one that matters most: the evaluator runs the compose file as-is and never edits it, so if it does not resolve against .env.example alone, it is not shippable.

If --check reports drift you have local edits to generated files. Discard them; the generators are the source of truth.

The generated plugin-layers/** Dockerfiles changed on 2026-07-27 (the permission normalisation moved into the payload carrier — see §3.1 rule 3). Any tree whose plugin-layers/ predates that must re-run stage-plugin-layers.py plan, which the commands above already do. A stale tree fails check; it does not silently ship the old shape.

2. Re-sign the plugin bundles — RELEASE BLOCKER

Services load their transports, analyzers and connectors from signed plugin bundles, and signature enforcement is on. The bundles produced by the dev tooling are signed with development keys:

grep -o '"keyId": "[^"]*"' devops/plugins/scanner/base/*/manifest.json | sort -u
# stella-scanner-plugin-dev-...

Authority’s signature provider is likewise configured as offline-dev-rsa-sha256, and the router-transport signer in build-service-publish.sh documents itself as DEV-ONLY.

Re-sign every bundle with the release key and refresh the matching trust roots under devops/etc/certificates/trust-roots/plugins/**.

Verify: no -dev- key id remains in any shipped manifest.json, and each module’s trust root contains the release public key.

Publishing without this ships a product whose signed-plugin integrity model is anchored to a key that was never meant to leave a laptop. Do not skip it.

3. Build

./tools/build-and-publish.sh --version "$STELLA_VERSION" --channel "$STELLA_BUNDLE_CHANNEL"

Nothing is pushed. This builds every service, bakes the signed plugin bundles and the console UI into the images, and tags them at registry.stella-ops.org/stellaops/<repo>:<version>.

Expect a long run — dozens of .NET publishes.

Watch for no local image for stellaops/<repo>. That service was never built. It will be missing from the release. Build it (devops/docker/build-service-publish.sh <repo>) and re-run.

Verify:

docker images "registry.stella-ops.org/stellaops/*" --format '{{.Repository}}:{{.Tag}}' | wc -l
devops/build/audit-pluginized-image.sh registry.stella-ops.org/stellaops/scanner-worker:$STELLA_VERSION

The count should match the image count the script reported at the start. The audit should pass. It permits signed documentation artefacts under the gateway’s inert /app/wwwroot static tree, while still rejecting plugin signatures or assemblies anywhere outside /app/plugins. Set FAIL_ON_PLUGIN_ROOT_PAYLOAD=1 for the stricter mounted-mode audit.

3.1 Packaging rules — what a shippable image looks like

The download is the product’s first impression, and it is the one number an evaluator sees before anything else works. These rules are enforced by tooling, not by review, because the failure mode is silent: an image that carries five superseded copies of itself still runs perfectly.

The rules.

1. A service image is built FROM a clean base, never from its own previous output. devops/docker/build-service-publish.sh resolves stellaops/service-base:<flavour>-<rev> (built by build-service-base.sh from mcr.microsoft.com/dotnet/aspnet:10.0-noble). The flavours are default, ldap (Authority’s OpenLDAP interop) and go (scanner-worker’s Go SSA call-graph toolchain). A base never depends on anything a service build produces, so the layer count is a constant rather than a function of how many times the service has been rebuilt.
2. The payload is written exactly once. The publish overlay adds two non-empty layers — healthcheck.sh and /app. File modes are applied in a throwaway build stage; a chmod in the runtime stage would copy up every file it touched and write the payload a second time.
3. The signed plugin bundles are written exactly once. The payload carrier image normalises permissions to root-owned 0555/0444 in a discarded stage, and each service’s release layer just COPY --froms its subtrees, which preserves mode and ownership.
4. Ports and identity are stated, never inherited. APP_BINARY, APP_PORT and APP_URLS are passed in per service; nothing survives only because a previous image happened to carry it.

The budget — devops/docker/image-budget.env, one reviewed file:

Size is measured by summing docker history layer sizes, not docker image inspect .Size: with the containerd image store the latter reports compressed bytes and with overlay2 uncompressed, so it is not comparable between a laptop and a CI runner.

Adding an allowance row is a review decision, not a workaround. Say what the bytes are and where you measured them.

How to verify, any time:

# one image, or the whole local estate
devops/docker/check-image-budget.sh registry.stella-ops.org/stellaops/authority:$STELLA_VERSION
devops/docker/check-image-budget.sh --all --registry registry.stella-ops.org
devops/docker/check-image-budget.sh --all --json     # for CI

Exit 0 means in budget; exit 1 means at least one breach; exit 2 is a usage or environment problem. This is the command to wire into a CI packaging job — it needs only a docker daemon that can see the images and mutates nothing.

The same check runs automatically in three places, so a breach cannot reach a registry unnoticed:

• after every build-service-publish.sh build, on the image just produced;
• in build-plugin-layers.sh, on the base images before layering and on the layered images afterwards;
• in build-and-publish.sh as stage 3b, over the full release set, before the push.

When the budget check fails.

STELLA_ALLOW_IMAGE_BLOAT=1 downgrades the release-tooling gates to warnings. It exists for debugging a single stage. Never use it for a release — it was a warning that let 67 GB of dead layers accumulate unchallenged in the first place.

Historical note, so nobody re-derives this the hard way. Until 2026-07-27 build-service-publish.sh passed BASE_IMAGE=stellaops/<svc>:dev — the tag it was about to overwrite. Measured with docker history on the reference host just before the fix:

Rebuilt from the clean base, from the identical publish output: authority 10 layers / 290 MB, platform 10 layers / 365 MB. The /app file tree and its modes, the entrypoint, workdir, user, exposed port and healthcheck are byte-identical to the images they replace.

4. Push and sign

This is the irreversible step. A tag, once pushed, is public.

./tools/build-and-publish.sh --version "$STELLA_VERSION" --channel "$STELLA_BUNDLE_CHANNEL" \
  --push --sign

This pushes every image, reads back each registry-assigned digest, writes release-manifest.yaml with those digests and the per-file checksums, generates docker-compose.pinned.yml (digest-pinned, the reproducible form), and cosign-signs the manifest and each image.

Verify:

python tools/verify-bundle.py --require-digests --require-signature
cosign verify-blob --insecure-ignore-tlog=true --key release-signing.pub \
  --signature release-manifest.yaml.sig release-manifest.yaml

Both must pass. verify-bundle.py fails closed: a missing checksum block, an entry with no digest, or an absent signature is an error, not a warning.

5. Package and stage the download

The archive must extract into a directory named after the release, because every published command sequence says tar xzf … then cd stellaops-bundle-$STELLA_VERSION. tar … -C bundle . does not do that — it writes members as ./…, so extraction dumps 248+ files into the evaluator’s working directory and there is nothing to cd into. Stage a correctly named directory first.

tar also does not honour .gitignore, and the release machine’s devops/release/bundle/ accumulates per-installation state after any local test install — .env, secrets/, generated *.pem / *.pfx. Those are exactly the files devops/release/bundle/.gitignore lists, and shipping them would publish your own signing keys and admin password. Exclude them explicitly.

Scope the key exclusions to the directories install.sh writes into. A blanket --exclude='*.pem' is wrong and was a hard blocker until 2026-07-30: three plugin trust roots — config/trust-roots/plugins/execution/…, config/trust-roots/plugins/router/…, config/trust-roots/router-plugins/… — are .pem files that release-manifest.yaml checksums, so excluding them produced an archive that install.sh:169-171 aborts on with three missing: warnings. Key-based signing has no Fulcio certificate, so there is no release-manifest.yaml.pem to package.

The exclusion list below is the same rule write-manifest.py applies when it decides what to checksum (CHECKSUM_EXCLUDE_DIRS × CHECKSUM_EXCLUDE_SUFFIXES, plus the whole operator-owned config/etc/exportcenter/keys/ directory). Those lists must stay in agreement — when packaging excludes a file the manifest checksums, the installer fails closed on a bundle that is not actually corrupt. Generated private files live under config/etc/authority/keys/, config/etc/scanner/keys/, config/etc/release-orchestrator/keys/ and secrets/ (which is excluded wholesale). Operator-owned Export custody lives under config/etc/exportcenter/keys/, which is also excluded wholesale.

cd "$(git rev-parse --show-toplevel)"
STAGE="tmp/build/dist"
NAME="stellaops-bundle-${STELLA_VERSION}"

rm -rf "$STAGE/$NAME"
mkdir -p "$STAGE/$NAME"
tar -c -C devops/release/bundle \
    --exclude=.env --exclude=.env.local --exclude='.env.*.local' \
    --exclude=./secrets --exclude=./airgap-import --exclude=./offline \
    --exclude=./plugins --exclude=__pycache__ --exclude='*.tar.gz' \
    --exclude='./config/etc/authority/keys/*.pem' \
    --exclude='./config/etc/authority/keys/*.pfx' \
    --exclude='./config/etc/scanner/keys/*.pem' \
    --exclude='./config/etc/release-orchestrator/keys/*.pem' \
    --exclude='./config/etc/exportcenter/keys' \
    . | tar -x -C "$STAGE/$NAME"

tar czf "$STAGE/${NAME}.tar.gz" -C "$STAGE" "$NAME"

Verify the archive before staging it — these are the checks that would have caught all three defects above. Run all of them; each catches a different one.

# 1. structure — must be stellaops-bundle-<version>/
tar tzf "$STAGE/${NAME}.tar.gz" | head -1

# 2. leak — per-installation state must not be in the archive.
#    Matches the paths install.sh actually generates, NOT every *.pem: the
#    plugin trust roots under config/trust-roots/ are legitimate bundle content.
tar tzf "$STAGE/${NAME}.tar.gz" \
  | grep -E '(^|/)(\.env$|secrets/|config/etc/exportcenter/keys/|config/etc/(authority|scanner|release-orchestrator)/keys/[^/]+\.(pem|pfx)$)' \
  && { echo 'ABORT: per-installation state in the archive'; exit 1; }

# 3. completeness — every file the manifest checksums must be in the archive.
#    Catches the inverse failure: an over-broad exclusion that produces an
#    archive install.sh:169-171 rejects as corrupt.
python - "$STAGE/${NAME}.tar.gz" "$NAME" <<'PY'
import sys, tarfile, re
archive, name = sys.argv[1], sys.argv[2]
members = {m.name for m in tarfile.open(archive)}
want = re.findall(r'^  - path: "(.+)"$',
                  open('devops/release/bundle/release-manifest.yaml',
                       encoding='utf-8').read(), re.M)
missing = [p for p in want if f"{name}/{p}" not in members]
print(f"{len(want) - len(missing)}/{len(want)} manifest files present")
if missing:
    for p in missing[:20]:
        print("  MISSING:", p)
    raise SystemExit("ABORT: archive is missing files the manifest checksums")
PY

Then stage the upload tree:

python devops/release/site/build-site-tree.py --version "$STELLA_VERSION" --dist "$STAGE"

--dist now defaults to this same tmp/build/dist/, so passing it is optional. It is kept explicit here because the two paths must agree and a silent default is how they drifted apart the last time.

build-site-tree.py assembles the exact directory to upload and exits non-zero listing anything missing, so an incomplete release cannot be staged by accident. It produces both the versioned directory and the latest/ copy the site links to, under the repository root’s tmp/build/site/.

Verify — from the staged tree, as a stranger would:

cd tmp/build/site/releases/$STELLA_VERSION
sha256sum --check SHA256SUMS

6. Upload

The download site is not S3. It is the static Nginx vhost on deployment@ssh.stella-ops.org with host document root:

/home/deployment/stella-ops.org/sites/get.stella-ops.org/www

Copy tmp/build/site/releases/$STELLA_VERSION/ and latest/ to a unique incoming directory beside www/, run sha256sum --check SHA256SUMS in both remote directories, then move them into www/releases/ only after both checks pass. The versioned destination must not already exist. Do not use sync --delete: published version directories are immutable release history.

Verify from outside your network — a fresh shell, no local caches:

curl -fsSLO https://get.stella-ops.org/releases/$STELLA_VERSION/stellaops-bundle-$STELLA_VERSION.tar.gz
curl -fsSL  https://get.stella-ops.org/releases/$STELLA_VERSION/SHA256SUMS | sha256sum --check --ignore-missing
curl -fsSI  https://get.stella-ops.org/releases/latest/docker-compose.yml | head -1

The latest/ request must return 200, not a redirect — the documented flow is a bare curl <base>/<file>.

7. Before you announce

Do a real install on a machine that has never seen this product: no local images, no cached layers, no .env.

tar xzf stellaops-bundle-$STELLA_VERSION.tar.gz
cd stellaops-bundle-$STELLA_VERSION
./install.sh

• [ ] Images pull anonymously — no docker login needed
• [ ] The stack converges and the installer prints a console URL and a generated admin password
• [ ] The console loads and that password signs you in
• [ ] docker compose ps shows every service healthy
• [ ] Nothing required editing docker-compose.yml

Then check the claims:

• [ ] Site download links resolve, versioned and latest/
• [ ] Quoted free-tier limits match LICENSE
• [ ] Published system requirements match what you just observed
• [ ] Release notes exist under docs/releases/

If the console does not load, start at the gateway: docker compose logs router-gateway. The UI is baked into that image at /app/wwwroot — there is no separate console service and no build step.

8. Rolling back

Nothing is deleted. Registry tags and published files are immutable history; rolling back means pointing people at the previous version, not erasing this one.

1. Repoint latest/— re-run build-site-tree.py and the upload for the previous version. This is the only step most rollbacks need: it is what new evaluators fetch.
2. Leave the version directory and image tags in place. Someone is mid-install against them. Removing a tag someone is pulling turns a bad release into a broken one.
3. Publish the reason in docs/releases/ and on the site. A release withdrawn silently is indistinguishable from one that was never noticed.
4. If the problem is a compromised signing key, that is not a rollback — follow the key-rotation path, announce out-of-band, and do not sign the new key with the old one.

Existing installations are unaffected: they run digests they already pulled.

Related

• devops/docker/image-budget.env — the packaging budget (§3.1), and the only place an allowance may be granted
• devops/docker/check-image-budget.sh — the checker; --all --json is the CI entry point
• devops/docker/Dockerfile.service-base — the clean base every service image is built FROM, and why
• release-checklist.md — reviewer guardrails for a production tag
• devops/release/SITE-INTEGRATION.md — what the website must say, and what it must stop saying
• devops/release/bundle/README.md — the evaluator-facing document
• docs-archive/implplan/SPRINT_20260727_001_DevOps_release_bundle_registry_publication.md — completed RC1 release execution record