13 · Release Engineering Playbook — Stella Ops

A concise, automation‑first guide describing how source code on main becomes a verifiably signed, air‑gap‑friendly release. It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).

0 Release Philosophy

• Fast but fearless – every commit on main must be releasable; broken builds break the build, not the team.
• Reproducible – anyone can rebuild byte‑identical artefacts with a single make release offline.
• Secure by default – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.
• Offline‑first – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.
• Non-Kubernetes deployment - Stella Ops does not ship or support Kubernetes manifests or Helm charts. Release packaging targets Docker Compose, Offline Kit, and signed release manifests only.

1 Versioning & Branching

• SemVer – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.
• Release tags are signed (git tag -s) with the Stella Ops GPG key (0x90C4…).

2 CI/CD Overview (GitLab CI + GitLab Runner)

graph LR
  A[push / MR] --> Lint
  Lint --> Unit
  Unit --> Build
  Build --> Test-Container
  Test-Container --> SBOM
  SBOM --> Sign
  Sign --> Publish
  Publish --> E2E
  Publish --> Notify

Pipeline Stages

All stages run in parallel where possible; max wall‑time < 15 min.

Implementation note. .gitea/workflows/release.yml executes ops/devops/release/build_release.py to build multi-arch images, attach CycloneDX SBOMs and SLSA provenance with Cosign, and emit out/release/release.yaml for downstream packaging (Compose and Offline Kit). The build-test-deploy workflow also runs python ops/devops/release/test_verify_release.py so release verifier regressions fail fast during every CI pass.

After build/publish produces final container digests, release engineering must materialize stable and air-gap release-manifest authority from a local artifact instead of resolving tags. First produce that artifact from local build outputs:

python devops/release/produce_image_digests.py \
  --channel-source stable=out/release/stable/artifacts/metadata \
  --channel-source airgap=out/release/airgap/artifacts/metadata \
  --manifest stable=devops/releases/2025.09-stable.yaml \
  --manifest airgap=devops/releases/2025.09-airgap.yaml \
  --infrastructure-source out/release/infrastructure-digests.json \
  --output out/release/image-digests.json

Component digests must come from Docker Buildx --metadata-file output written by successful publish jobs. Already digest-pinned PostgreSQL and Valkey entries can be carried from the local release manifests. RustFS and any other tag-only infrastructure image must be supplied by a local infrastructure source captured from its real mirror/build result. The output artifact is JSON with manifests.<channel>.components and manifests.<channel>.infrastructure mappings to final image@sha256:<digest> references. Apply and validate it offline:

python devops/release/apply_image_digests.py \
  --artifact out/release/image-digests.json \
  --manifest devops/releases/2025.09-stable.yaml \
  --manifest devops/releases/2025.09-airgap.yaml

python devops/release/check_release_manifest.py \
  --manifest devops/releases/2025.09-stable.yaml \
  --manifest devops/releases/2025.09-airgap.yaml

The patcher refuses tag-only values, tag-plus-digest values, missing required components, repository/name mismatches, and partial stable/air-gap updates before writing either manifest.

This tooling is retained as release-manifest authority for non-Kubernetes packaging; it is not Helm support.

Release-guide TODOs.

• TODO(release-digests): update the required component/infrastructure digest list when release payload composition changes.
• TODO(release-schema): update the release-manifest schema notes when bundle metadata or evidence-pack schemas change.
• TODO(model): current setup guidance pins microsoft/bitnet-b1.58-2B-4T as the default small ternary advisory model. Update this release guide and model-selection docs before changing the default to a custom/larger Stella Ops model.
• TODO(provider): update provider-pack guidance when crypto/evidence provider packs are selected and legally cleared.
• TODO(dora-taxonomies): re-pin DORA taxonomy/schema package versions during every release branch hardening window. Do not carry the previous release’s DORA taxonomy pins forward without a fresh source/version/hash check and offline validation evidence.

For local tests only, developers may generate a simulated digest artifact:

python devops/release/produce_image_digests.py \
  --simulate-buildx-fixture-for-tests \
  --manifest stable=devops/releases/2025.09-stable.yaml \
  --manifest airgap=devops/releases/2025.09-airgap.yaml \
  --output out/release/image-digests.test-only.json

That artifact carries testOnly: true and sourceKind: simulated-buildx-fixture. Production apply and validation reject it by default. The patcher accepts it only with --allow-test-only-artifact-write-for-tests, stamps output manifests as test-only, and those manifests still fail check_release_manifest.py and runtime posture validation unless the corresponding --allow-...-for-tests flags are used. Simulated digests are never stable or air-gap release authority.

Compliance Provider And Schema Asset Intake

External compliance assets are release inputs only after approved offline intake. Release engineering must not download provider packs, taxonomies, XSDs, or trust lists during a release build.

Current pinned-but-not-ingested assets:

Any ingested external asset must have a recorded source, immutable local path, manifest hash, license/notice review, and offline validation fixture. If the asset cannot be redistributed, the release must point to the internal sealed asset location and describe how customer deployments provide it without network fetches.

DORA pins are release-scoped. During release hardening, release engineering must verify whether the EBA DORA incident-reporting and RoI taxonomy packages have changed, record the selected version and source hash in the release evidence pack, and rerun offline schema-validation fixtures. A release must not publish DORA conformance claims from stale taxonomy pins that were not rechecked for that release.

Packaging support note: Kubernetes and Helm artifacts are not part of the current supported Stella Ops release model. Legacy Helm/Kubernetes references in release documentation are not release authority and must be cleaned up before publishing the next operator-facing release guide.

3 Container Image Strategy

Images are --label org.opencontainers.image.source=git.stella-ops.ruand include SBOMs generated at build time.

4 📌 Offline Update Kit (OUK) Build & Distribution

Purpose – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.

4.1 CLI Tool

Go binary ouk lives in src/Tools/ouk/.

ouk fetch \
  --nvd --osv \
  --trivy-db --date $(date -I) \
  --output ouk-$(date +%Y%m%d).tar.gz \
  --sign cosign.key

4.2 Pipeline Hook

• Runs on first Friday each month (cron).
• Generates tarball, signs it, uploads to GitLab Release asset.
• SHA‑256 + signature published alongside.
• Release job must emit out/release/debug/ with debug-manifest.json and .sha256 so ops/offline-kit/mirror_debug_store.py can mirror symbols into the Offline Kit (see DEVOPS-REL-17-004).

4.3 Activation Flow (runtime)

1. Admin uploads .tar.gz via UI → Settings → Offline Updates (OUK).
2. Backend verifies Cosign signature & digest.
3. Files extracted into var/lib/stella/db.
4. Valkey caches invalidated; Dashboard “Feed Age” ticks green.
5. Audit event ouk_update stored.

4.4 Token Detail

client.jwt placed under /root/ inside the tarball. CI job fails if token expiry < 29 days (guard against stale caches).

5 Artifact Signing & Transparency

Rekor integration is TODO – once the internal Rekor mirror is online (StellaOpsAttestor) a post‑publish job will submit transparency log entries.

6 Release Checklist

1. CI pipeline green.
2. Bump VERSION file.
3. Tag git tag -s X.Y.Z -m "Release X.Y.Z" & push.
4. GitLab CI auto-publishes images, release manifests, and offline bundles.
5. Draft GitLab Release Notes using src/Tools/release-notes-gen.
6. Verify SBOM attachment with stella sbom verify stella/backend:X.Y.Z.
7. Run the release verifier locally if CI isn’t available (mirrors the workflow step): python ops/devops/release/test_verify_release.py
8. Verify reproducibility – rebuild and compare checksums:

   export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)
   make release
   sha256sum dist/* | diff - out/release/SHA256SUMS
   

9. Generate Release Evidence Pack – trigger evidence pack workflow:

   gh workflow run release-evidence-pack.yml \
     -f version=X.Y.Z \
     -f release_tag=vX.Y.Z
   

10. Self-verify evidence pack – extract and run verify.sh:

    tar -xzf stella-release-X.Y.Z-evidence-pack.tgz
    cd stella-release-X.Y.Z-evidence-pack
    ./verify.sh --verbose
    

11. Mirror the release debug store into the Offline Kit staging tree and re-check the manifest:

./ops/offline-kit/mirror_debug_store.py \
  --release-dir out/release \
  --offline-kit-dir out/offline-kit
jq '.artifacts | length' out/offline-kit/debug/debug-manifest.json
readelf -n /app/... | grep -i 'Build ID'

Validate that the hash from readelf matches the .build-id/<aa>/<rest>.debug path created by the script. 12. Smoke-test OUK tarball in offline lab. 13. Announce in #stella-release Mattermost channel.

7 Hot‑fix Procedure

• Branch from latest tag → hotfix/X.Y.Z+1-hf1.
• Apply minimal patch, add regression test.
• CI pipeline (with reduced stages) must pass.
• Tag X.Y.Z+1.
• Publish only containers and the signed release manifest/offline bundle entries required by the fix; OUK not rebuilt unless feed content changed.
• Cherry‑pick back to main.

8 Deprecation & End‑of‑Life Policy

At least 6 months notice; removal requires major version bump.

9 📌 Non‑Commercial Usage Rules (English canonical)

1. Free for internal security assessments (company or personal).
2. SaaS resale / re-hosting prohibited without prior written consent (policy requirement; not a license restriction).
3. If you distribute a fork with UI or backend modifications you must:
   • Include the LICENSE and NOTICE files.
   • Mark modified files with prominent change notices.
   • Retain the original Stella Ops attribution in UI footer and CLI --version.
4. All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).
5. Deployments in state‑regulated or classified environments must obeyapplicable local regulations governing cryptography and software distribution.

10 Best Practices Snapshot 📌

• SBOM‑per‑image → attach at build time; store as OCI artifact for supply‑chain introspection.
• Provenance flag (--provenance=true) in BuildKit fulfils SLSA 2 requirement.
• Use multi‑arch, reproducible builds (SOURCE_DATE_EPOCH pins timestamps).
• All pipelines enforce Signed‑off‑by (DCO); CI fails if trailer missing.
• cosign policy ensures only images signed by the project key run in production.

11 Contributing to Release Engineering

• Fork & create MR to infra/release-*.
• All infra changes require green integration-e2e-offlinejob.
• Discuss larger infra migrations in #sig-release Mattermost; decisions recorded in ADR/ folder.

12 Change Log (high‑level)

(End of Release Engineering Playbook v1.1)