13 · Release Engineering Playbook — Stella Ops

A concise, automation‑first guide describing how source code on main becomes a verifiably signed, air‑gap‑friendly release.
It is opinionated for offline use‑cases and supply‑chain security (SLSA ≥ level 2 today, aiming for level 3).

0 Release Philosophy

Fast but fearless – every commit on main must be releasable; broken builds break the build, not the team.
Reproducible – anyone can rebuild byte‑identical artefacts with a single make release offline.
Secure by default – every artefact ships with a SBOM, Cosign signature and (future) Rekor log entry.
Offline‑first – all dependencies are vendored or mirrored into the internal registry; no Internet required at runtime.

1 Versioning & Branching

Branch	Purpose	Auto‑publish?
`main`	Always‑green development trunk	`nightly-*` images
`release/X.Y`	Stabilise a minor line	`stella:X.Y-rcN`
Tags	`X.Y.Z` = SemVer	`stella:X.Y.Z`, OUK tarball, Helm chart

SemVer – MAJOR for breaking API/CLI changes, MINOR for features, PATCH for fixes.
Release tags are signed (git tag -s) with the Stella Ops GPG key (0x90C4…).

2 CI/CD Overview (GitLab CI + GitLab Runner)

graph LR
  A[push / MR] --> Lint
  Lint --> Unit
  Unit --> Build
  Build --> Test-Container
  Test-Container --> SBOM
  SBOM --> Sign
  Sign --> Publish
  Publish --> E2E
  Publish --> Notify

Pipeline Stages

Stage	Key tasks
Lint	ESLint, golangci‑lint, hadolint, markdown‑lint.
Unit	`dotnet test`, `go test`, Jest UI tests.
Quota unit‑tests 🏷	Validate QuotaService logic: reset at UTC, 5 s vs 60 s waits, header correctness.
Build	Multi‑arch container build (`linux/amd64`, `linux/arm64`) using BuildKit + `--provenance` 📌.
Test‑Container	Spin up compose file, run smoke APIs.
SBOM 📌	Invoke StellaOps.SBOMBuilder to generate SPDX JSON + attach `.sbom` label to image.
Sign	Sign image with Cosign (`cosign sign --key cosign.key`).
Publish	Push to `registry.git.stella-ops.org`.
E2E	Kind‑based Kubernetes test incl. Zastava DaemonSet; verify sub‑5 s scan SLA.
Notify	Report to Mattermost & GitLab Slack app.
OfflineToken	Call `JwtIssuer.Generate(exp=30d)` → store `client.jwt` artefact → attach to OUK build context

All stages run in parallel where possible; max wall‑time < 15 min.

3 Container Image Strategy

Image	Registry Tag	Contents
backend	`stella/backend:{ver}`	ASP.NET API, plugin loader.
ui	`stella/ui:{ver}`	Pre‑built Angular SPA.
runner-trivy	`stella/runner-trivy:{ver}`	Trivy CLI + SPDX/CycloneDX 🛠.
runner-grype	`stella/runner-grype:{ver}`	Optional plug‑in scanner.
🏷️ StellaOps.Registry 📌	`stella/registry:{ver}`	Scratch image embedding Docker Registry v2 + Cosign policy controller.
🏷️ StellaOps.MutePolicies 📌	`stella/policies:{ver}`	Sidecar serving policy bundles.
🏷️ StellaOps.Attestor 📌	`stella/attestor:{ver}`	SLSA provenance & Rekor signer (future).

Images are --label org.opencontainers.image.source=git.stella-ops.ru and include SBOMs generated at build time.

4 📌 Offline Update Kit (OUK) Build & Distribution

Purpose – deliver updated CVE feeds & Trivy DB to air‑gapped clusters.

4.1 CLI Tool

Go binary ouk lives in tools/ouk/.

ouk fetch \
  --nvd --osv \
  --trivy-db --date $(date -I) \
  --output ouk-$(date +%Y%m%d).tar.gz \
  --sign cosign.key

4.2 Pipeline Hook

Runs on first Friday each month (cron).
Generates tarball, signs it, uploads to GitLab Release asset.
SHA‑256 + signature published alongside.

4.3 Activation Flow (runtime)

Admin uploads .tar.gz via UI → Settings → Offline Updates (OUK).
Backend verifies Cosign signature & digest.
Files extracted into var/lib/stella/db.
Redis caches invalidated; Dashboard “Feed Age” ticks green.
Audit event ouk_update stored.

4.4 Token Detail

client.jwt placed under /root/ inside the tarball. CI job fails if token expiry < 29 days (guard against stale caches).

5 Artifact Signing & Transparency

Artefact	Signer	Tool
Git tags	GPG (`0x90C4…`)	`git tag -s`
Containers	Cosign key pair	`cosign sign`
Helm Charts	prov file	`helm package --sign`
OUK tarballs	Cosign	`cosign sign-blob`

Rekor integration is TODO – once the internal Rekor mirror is online (StellaOpsAttestor) a post‑publish job will submit transparency log entries.

6 Release Checklist

CI pipeline green.
Bump VERSION file.
Tag git tag -s X.Y.Z -m "Release X.Y.Z" & push.
GitLab CI auto‑publishes images & charts.
Draft GitLab Release Notes using tools/release-notes-gen.
Verify SBOM attachment with stella sbom verify stella/backend:X.Y.Z.
Smoke‑test OUK tarball in offline lab.
Announce in #stella-release Mattermost channel.

7 Hot‑fix Procedure

Branch from latest tag → hotfix/X.Y.Z+1-hf1.
Apply minimal patch, add regression test.
CI pipeline (with reduced stages) must pass.
Tag X.Y.Z+1.
Publish only container + Helm chart; OUK not rebuilt.
Cherry‑pick back to main.

8 Deprecation & End‑of‑Life Policy

Feature	Deprecation notice	Removal earliest
Legacy CSV policy import	2025‑10‑01	2026‑04‑01
Docker v1 Registry auth	2025‑12‑01	2026‑06‑01
In‑image Trivy DB	2025‑12‑15	2026‑03‑15

At least 6 months notice; removal requires major version bump.

9 📌 Non‑Commercial Usage Rules (English canonical)

Free for internal security assessments (company or personal).
SaaS resale / re‑hosting prohibited without prior written consent (AGPL §13).
If you distribute a fork with UI or backend modifications you must:
- Publish the complete modified source code.
- Retain the original Stella Ops attribution in UI footer and CLI --version.
All third‑party dependencies remain under their respective licences (MIT, Apache‑2.0, ISC, BSD).
Deployments in state‑regulated or classified environments must obeyapplicable local regulations governing cryptography and software distribution.

10 Best Practices Snapshot 📌

SBOM‑per‑image → attach at build time; store as OCI artifact for supply‑chain introspection.
Provenance flag (--provenance=true) in BuildKit fulfils SLSA 2 requirement.
Use multi‑arch, reproducible builds (SOURCE_DATE_EPOCH pins timestamps).
All pipelines enforce Signed‑off‑by (DCO); CI fails if trailer missing.
cosign policy ensures only images signed by the project key run in production.

11 Contributing to Release Engineering

Fork & create MR to infra/release-*.
All infra changes require green integration-e2e-offline job.
Discuss larger infra migrations in #sig-release Mattermost; decisions recorded in ADR/ folder.

12 Change Log (high‑level)

Version	Date	Note
v2.1	2025‑07‑15	Added OUK build/publish pipeline, internal registry image (`StellaOps.Registry`), non‑commercial usage rules extraction, SBOM stage, BuildKit provenance.
v2.0	2025‑07‑12	Initial open‑sourcing of Release Engineering guide.
v1.1	2025‑07‑09	Fixed inner fencing; added retention policy
v1.0	2025‑07‑09	Initial playbook

(End of Release Engineering Playbook v1.1)