Release pipelines

Reference for Stella Ops release automation — suite releases, per-module publishing, bundle and per-service releases, rollback, and environment promotion. It is for release engineers and operators who cut, sign, and promote builds. Read the status note below first: most of these pipelines currently live in .gitea/workflows-archived/, so this page is the design-of-record — verify the YAML before relying on any specific job, input, or registry name.

Status (reconciled 2026-05-30 against .gitea/): The release, module-publish, rollback, and promotion pipelines described below currently live in .gitea/workflows-archived/, not in the active .gitea/workflows/ directory. On 2026-02-01 all 118 production workflows were archived so active CI could be decoupled from the catalog while the local-CI scaffolding was validated (see .gitea/README.md). A subset of CI/test/e2e workflows has since been re-activated, but the release/publish/rollback/promote workflows remain archived — they are runnable locally with act against the archived path and can be restored by mv-ing the file back into .gitea/workflows/. Treat this page as the design-of-record for those pipelines; verify against the YAML in .gitea/workflows-archived/ before relying on any specific job, input, or registry name.

Release Strategy Overview

Stella Ops uses a dual-versioning strategy:

1. Suite Releases - Ubuntu-style YYYY.MM versioning with codenames
2. Module Releases - Semantic versioning MAJOR.MINOR.PATCH per module

Release Channels

The edge / stable / lts channel names are real: they are the channel choice inputs on release-suite.yml and release.yml, and become container image tags (:<channel>). On a suite-* tag push the channel is derived from the calendar month (04 → lts, 10 → stable).

The “every merge to main / bi-weekly / quarterly” cadences are intent, not an implemented automation: no workflow auto-publishes an Edge release on merge to main. Releases are produced by pushing a suite-* / v* / module-*-v* / service-*-v* tag or by manual workflow_dispatch.

Suite Release Pipeline

Trigger

# Tag-based trigger (month MUST be 04 or 10; codename suffix optional)
git tag suite-2026.04
git push origin suite-2026.04

# Or manual trigger via Gitea Actions UI
# Workflow: release-suite.yml (currently in .gitea/workflows-archived/)
# Inputs: version, codename, channel, skip_tests, dry_run

Versioning constraint (source: release-suite.yml parse-tag/validate): the workflow only accepts YYYY.MM where MM is 04 or 10 — arbitrary months are rejected with an error. On a tag push the channel is derived from the month (04 → lts, 10 → stable); the channel input is only honoured on manual workflow_dispatch. The codename defaults to TBD if omitted.

Workflow: release-suite.yml

┌─────────────────────────────────────────────────────────────────┐
│                     SUITE RELEASE PIPELINE                       │
│                                                                 │
│  ┌──────────────┐                                               │
│  │  parse-tag   │ (if triggered by tag push)                    │
│  │  or validate │ (if triggered manually)                       │
│  └──────┬───────┘                                               │
│         │                                                       │
│         ▼                                                       │
│  ┌──────────────┐                                               │
│  │  test-gate   │ (optional, skipped with skip_tests=true)      │
│  └──────┬───────┘                                               │
│         │                                                       │
│    ┌────┴────────────────────────────────────────┐              │
│    │                 BUILD PHASE                  │              │
│    │                                              │              │
│    │  ┌─────────────────┐  ┌─────────────────┐   │              │
│    │  │  build-modules  │  │ build-containers│   │              │
│    │  │  (9 in parallel)│  │  (9 in parallel)│   │              │
│    │  └─────────────────┘  └─────────────────┘   │              │
│    │                                              │              │
│    │  ┌─────────────────┐  ┌─────────────────┐   │              │
│    │  │   build-cli     │  │   build-helm    │   │              │
│    │  │ (5 platforms)   │  │                 │   │              │
│    │  └─────────────────┘  └─────────────────┘   │              │
│    │                                              │              │
│    └─────────────────────┬────────────────────────┘              │
│                          │                                       │
│                          ▼                                       │
│  ┌───────────────────────────────────────────────┐              │
│  │              release-manifest                  │              │
│  │  - SuiteRelease YAML manifest                  │              │
│  │  - SHA256SUMS over nupkg/tgz/zip/tar.gz        │              │
│  │  (SBOM + in-toto/DSSE provenance are produced  │              │
│  │   by the separate release.yml bundle pipeline) │              │
│  └───────────────────────┬────────────────────────┘              │
│                          │                                       │
│    ┌─────────────────────┴─────────────────────────┐            │
│    │                                               │            │
│    ▼                       ▼                       ▼            │
│ ┌──────────────┐  ┌──────────────┐  ┌──────────────┐           │
│ │  changelog   │  │  suite-docs  │  │  compose     │           │
│ │  generation  │  │  generation  │  │  generation  │           │
│ └──────────────┘  └──────────────┘  └──────────────┘           │
│                          │                                       │
│                          ▼                                       │
│  ┌───────────────────────────────────────────────┐              │
│  │              create-release                    │              │
│  │  - Assemble CLI/Helm/manifest/checksum assets  │              │
│  │  - gh release create suite-<version>           │              │
│  │  - Containers already pushed by build-containers│             │
│  │    (Cosign signing is a SEPARATE workflow —    │              │
│  │     release.yml / release-keyless-sign.yml)    │              │
│  └───────────────────────┬────────────────────────┘              │
│                          │                                       │
│                          ▼                                       │
│  ┌───────────────────────────────────────────────┐              │
│  │              commit-docs                       │              │
│  │  - git add docs/releases/<version> and commit  │              │
│  │    (suite docs + generated compose files)      │              │
│  └───────────────────────────────────────────────┘              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Suite Versioning

The image is tagged with three independent tags (:<version>, :<channel>, :latest) — not a single combined 2026.04-stable tag.

Modules Built

The build-modules/build-containers matrices in release-suite.yml build 9 modules. Each module is packed from its *.WebService (or Policy.Gateway) project and pushed to the Gitea container registry (git.stella-ops.org/stella-ops.org/<module>), not Docker Hub. NuGet packages are pushed to the Gitea NuGet feed (https://git.stella-ops.org/api/packages/stella-ops.org/nuget/index.json).

NOT IMPLEMENTED in release-suite.yml: SbomService, EvidenceLocker, and VexLens are not part of the suite build matrix (the previous version of this table listed them). All module versions are currently pinned to 1.0.0 in the generated SuiteRelease manifest. Tags are pushed as :<version>, :<channel>, and :latest.

CLI Platforms

The CLI is published from src/Cli/StellaOps.Cli/StellaOps.Cli.csproj for 5 runtimes as a self-contained single file (PublishSingleFile=true, PublishTrimmed=true, EnableCompressionInSingleFile=true). Each runtime is archived (.tar.gz for *nix, .zip for Windows) using the naming pattern stellaops-cli-<version>-<codename>-<runtime>.<ext>.

Module Release Pipeline

Trigger

# Tag-based trigger
git tag module-authority-v1.2.3
git push origin module-authority-v1.2.3

# Or manual trigger via Gitea Actions UI
# Workflow: module-publish.yml (currently in .gitea/workflows-archived/)
# Inputs: module, version, publish_nuget, publish_container, prerelease

The prerelease input is declared on the workflow but is not consumed by any job in module-publish.yml (it has no effect today). On a module-*-v* tag push, publish_nuget and publish_container both default to true.

Workflow: module-publish.yml

┌─────────────────────────────────────────────────────────────────┐
│                    MODULE PUBLISH PIPELINE                       │
│                                                                 │
│  ┌──────────────┐                                               │
│  │  parse-tag   │ Extract module name and version from tag      │
│  │  or validate │ Normalize manual inputs                       │
│  └──────┬───────┘                                               │
│         │                                                       │
│    ┌────┴────────────────────────────────────────┐              │
│    │                                              │              │
│    ▼                                              ▼              │
│  ┌──────────────┐                    ┌──────────────┐           │
│  │publish-nuget │ (if flag set)     │publish-cont. │           │
│  │              │                    │ (if flag set,│           │
│  │ - Restore    │                    │  module≠CLI) │           │
│  │ - Build      │                    │ - Buildx     │           │
│  │ - Pack       │                    │ - Build      │           │
│  │ - Push       │                    │ - Push       │           │
│  └──────────────┘                    │ (no in-line  │           │
│                                      │  scan/sign)  │           │
│                                      └──────────────┘           │
│                                              │                   │
│                 OR (if module=CLI)           │                   │
│                                              ▼                   │
│  ┌──────────────────────────────────────────────────┐           │
│  │                 publish-cli                       │           │
│  │  - Build for 5 platforms (matrix)                 │           │
│  │  - Self-contained single-file + trimmed           │           │
│  │    (NOT AOT; binaries are NOT code-signed here)   │           │
│  │  - Archive (.tar.gz / .zip)                        │           │
│  │  - Upload as workflow artifact                    │           │
│  └──────────────────────────────────────────────────┘           │
│                          │                                       │
│                          ▼                                       │
│  ┌──────────────────────────────────────────────────┐           │
│  │                    summary                        │           │
│  │  - Release notes                                  │           │
│  │  - Artifact links                                 │           │
│  │  - SBOM references                                │           │
│  └──────────────────────────────────────────────────┘           │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Module Tag Format

module-<name>-v<semver>

Examples:
  module-authority-v1.2.3
  module-scanner-v2.0.0
  module-cli-v3.1.0-beta.1

Available Modules

The module input is a fixed choice list in module-publish.yml. Names are title-cased in the workflow (Authority, Attestor, …); the tag form lower-cases the name (module-authority-v1.2.3). Container publishing is skipped when module == CLI; for CLI the multi-platform publish-cli job runs instead.

The previous version of this table listed sbomservice, evidencelocker, and vexlens, none of which appear in the module-publish.yml choice list. It also omitted Signer, Gateway, Scheduler, Orchestrator, TaskRunner, and Notify, which are present. Each non-CLI module maps to its *.WebService project (Policy maps to StellaOps.Policy.Gateway) via the case block in the publish-nuget job.

Bundle Release Pipeline

Trigger

# Tag-based trigger
git tag v2025.12.1
git push origin v2025.12.1

# Channel-specific tags
git tag v2025.12.0-edge
git tag v2025.12.0-stable
git tag v2025.12.0-lts

Workflow: release.yml

Triggered by a v* tag push or workflow_dispatch (inputs version, channel, calendar, push_images). Registry is registry.stella-ops.org(login via REGISTRY_USERNAME / REGISTRY_PASSWORD). The single build-release job:

• Runs fixture, NuGet-restore-source, and telemetry-storage validators.
• Publishes and Cosign-signs the Python analyzer plug-in (DEVOPS-REL-14-004), requiring COSIGN_KEY_REF or COSIGN_IDENTITY_TOKEN.
• Enforces the CLI parity gate (check_cli_parity.py).
• Runs build_release.py --version --channel --calendar --git-sha to produce the deterministic bundle (signed images, SBOMs, provenance, manifest, build-id debug store), then verify_release.py over out/release.
• Uploads stellaops-release-<version> and stellaops-debug-<version> artifacts.

The version override example in the workflow is 2025.10.0-edge; the calendar (YYYY.MM) is derived from the version when not supplied. SBOM (CycloneDX/SPDX) and in-toto/DSSE provenance are produced here by build_release.py, not in release-suite.yml.

Service Release Pipeline

MISSING from the previous version of this page. service-release.yml (in .gitea/workflows-archived/) is a distinct, per-service release pipeline listed under “Release Pipelines” in .gitea/README.md.

Service Versioning (source: .gitea/README.md)

Two forms of the release version, deliberately. StellaOpsReleaseVersion is bare (1.0.0-RC1) because MSBuild rejects a leading v — <Version>v1.0.0-RC1</Version> fails with 'v1.0.0-RC1' is not a valid version string. StellaOpsReleaseTag (v1.0.0-RC1) is the git-tag/image-tag/display form, and is the value migration 092_PlatformVersionSetting.sql seeds as the console-rendered PlatformVersion. The helpers above accept the bare form only and reject a leading v explicitly. A prerelease suffix is supported: until 2026-07-28 these helpers parsed X.Y.Z only, and read-service-version.sh silently truncated 1.0.0-RC1 to 1.0.0.

The current per-service version pins live in devops/releases/service-versions.json (also the manifest consulted by the rollback workflow).

Rollback Pipeline

Trigger

# Manual trigger only via Gitea Actions UI
# Workflow: rollback.yml (currently in .gitea/workflows-archived/)
# Inputs: environment, service, target_version, reason, skip_health_check

NOT FULLY IMPLEMENTED. In rollback.yml the Execute rollback, Health check, and notify steps are placeholders: rollback delegates to .gitea/scripts/release/rollback.sh only if [[ -f ... ]] (otherwise it prints a TODO and no-ops), kubectl/Helm context configuration is a # TODO, the health check is a placeholder loop, and notification is a stub (# TODO: Send to Slack/Teams/PagerDuty). The < 5 min SLA is an aspirational target, not a measured guarantee. When no target_version is given the previous version is resolved from devops/releases/service-versions.json.

Workflow: rollback.yml

┌─────────────────────────────────────────────────────────────────┐
│                      ROLLBACK PIPELINE                           │
│                     (SLA Target: < 5 min)                        │
│                                                                 │
│  ┌──────────────┐                                               │
│  │   validate   │ Verify inputs and permissions                 │
│  └──────┬───────┘                                               │
│         │                                                       │
│         ▼                                                       │
│  ┌──────────────┐                                               │
│  │ fetch-prev   │ Download previous version artifacts           │
│  │   version    │                                               │
│  └──────┬───────┘                                               │
│         │                                                       │
│         ▼                                                       │
│  ┌──────────────┐                                               │
│  │   execute    │ Calls rollback.sh IF present (else TODO/no-op);│
│  │   rollback   │ kubectl/Helm context config is a # TODO       │
│  └──────┬───────┘                                               │
│         │                                                       │
│         ▼                                                       │
│  ┌──────────────┐                                               │
│  │health-check  │ Placeholder loop (skipped if skip_health_check)│
│  └──────┬───────┘                                               │
│         │                                                       │
│         ▼                                                       │
│  ┌──────────────┐                                               │
│  │   notify     │ Stub: prints message + incident record only   │
│  │              │ (Slack/Teams/PagerDuty wiring is a # TODO)    │
│  └──────────────┘                                               │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Rollback Parameters

The previous version listed a dry_run boolean and a generic version field. The actual inputs are target_version and skip_health_check; there is no dry_run input on rollback.yml. The service choice list does not include SbomService/EvidenceLocker/VexLens.

Promotion Pipeline

Trigger

# Manual trigger only via Gitea Actions UI
# Workflow: promote.yml — "Promote Feedser (Manual)"
#           (currently in .gitea/workflows-archived/)
# Inputs: include_docs (boolean, default true), tag (string, default "latest")

CORRECTED. The actual promote.yml is a single staging → production artefact-promotion job, not a channel-promotion (from_environment / to_environment / version) pipeline. Its only inputs are include_docs and tag. .gitea/README.md describes it generically as “Environment promotion (dev/stage/prod)”, but the YAML hard-codes a staging source and a production target resolved from deployment secrets/vars.

Actual Promotion Flow (promote.yml)

  STAGING host                                  PRODUCTION host
 (rsync source)                                 (rsync target)
       │                                              ▲
       │  1. run fixture check                        │
       │  2. resolve STAGING_* + PRODUCTION_* creds    │
       │  3. rsync service bundle  ───────────────────┤  (after backup)
       │  4. rsync docs bundle (if include_docs) ──────┤
       │                                              │
       └──────────────► backup current production ────┘
                        content (keep last 5),
                        then publish over SSH

1. Inputs / fixtures — include_docs, tag; runs the offline-bundle fixture check (run-fixtures-check.sh).
2. Resolve credentials — staging and production host/user/path/key from STAGING_DEPLOYMENT_* / PRODUCTION_DEPLOYMENT_* secrets or vars (falling back to shared DEPLOYMENT_*). Missing required values fail the job.
3. Fetch from staging — rsync -az --delete the service bundle (and docs bundle if include_docs=true and a docs path is set) into a temp dir.
4. Backup production — timestamped backup of the current production content, retaining the most recent 5 backups.
5. Publish to production — rsync -az --delete the staged service bundle (and docs bundle) onto the production host.

The “Edge → Stable → LTS” channel-promotion model and the automated pre-flight/promotion/post-promotion checklist below are aspirational / roadmap, not implemented by promote.yml. They are retained as design intent.

Promotion Checklist (roadmap — not implemented in promote.yml)

1. Pre-Flight Checks

   • All tests passing in source environment
   • No critical vulnerabilities
   • Performance SLOs met
   • Documentation complete

2. Promotion Steps

   • Re-tag containers with new channel
   • Update Helm chart values
   • Deploy to target environment
   • Run smoke tests

3. Post-Promotion

   • Health check verification
   • Update release documentation
   • Notify stakeholders

Artifact Signing

Cosign Integration

Release artifacts are signed with Cosign. The keyless path uses Gitea’s OIDC issuer (https://git.stella-ops.org), not GitHub Actions. The registry is registry.stella-ops.org, not Docker Hub or GHCR. (release.yml additionally supports a keyed path via COSIGN_KEY_REF/COSIGN_PASSWORD for the analyzer plug-in.)

# Verify container signature (source: release-keyless-sign.yml)
# VERSION is the release version, e.g. 2025.10.0-edge
cosign verify \
  --certificate-identity "stella-ops.org/git.stella-ops.org:ref:refs/tags/v${VERSION}" \
  --certificate-oidc-issuer "https://git.stella-ops.org" \
  --require-rekor \
  registry.stella-ops.org/stella-ops.org/scanner:${VERSION}

CORRECTED. The previous example used ghcr.io/stellaops/scanner, --certificate-identity-regexp=".*github.com/stellaops.*", and the GitHub Actions OIDC issuer https://token.actions.githubusercontent.com. None of these match the codebase: there is no GHCR usage in the release pipelines, and keyless signing is bound to the self-hosted Gitea identity and Rekor.

Signature Artifacts

The release-suite.yml build-cli/publish-cli jobs do not emit .sig files for CLI binaries today; CLI signing is not wired into those workflows.

Release Artifacts

Per-Release Artifacts

Artifact Retention

Creating a Release

Suite Release

# 1. Ensure main is stable
git checkout main
git pull

# 2. Create and push tag
git tag suite-2026.04
git push origin suite-2026.04

# 3. Monitor release pipeline
# Gitea Actions → release-suite.yml

# 4. Verify artifacts
# - Check Gitea Releases page
# - Verify container images pushed
# - Validate SBOM and signatures

Module Release

# 1. Update module version
# Edit src/<Module>/version.txt or .csproj

# 2. Create and push tag
git tag module-authority-v1.2.3
git push origin module-authority-v1.2.3

# 3. Monitor release pipeline
# Gitea Actions → module-publish.yml

Hotfix Release

# 1. Create hotfix branch from release tag
git checkout -b hotfix/v2025.12.1 v2025.12.0

# 2. Apply fix
# ... make changes ...
git commit -m "Fix: critical security issue"

# 3. Create hotfix tag
git tag v2025.12.1
git push origin hotfix/v2025.12.1 v2025.12.1

# 4. Fast-track through pipeline
# Workflow will run with reduced test scope

Troubleshooting Releases

Release Pipeline Failed

1. Check build logs - Gitea Actions → failed job
2. Verify tag format - Must match expected pattern
3. Check secrets - Registry credentials, signing keys
4. Review test failures - May need to skip with skip_tests=true

Container Not Published

1. Check registry authentication - release-suite.yml / module-publish.yml log in to the Gitea registry with GITEA_TOKEN; release.yml uses REGISTRY_USERNAME / REGISTRY_PASSWORD against registry.stella-ops.org. (There is no REGISTRY_TOKEN secret.)
2. Verify image name - Check for typos in workflow
3. Check rate limits - May need to wait and retry
4. Confirm the workflow is active - the release/publish workflows live in .gitea/workflows-archived/; restore to .gitea/workflows/ before expecting them to run on push.

Signature Verification Failed

1. Check Sigstore/Rekor availability - May have temporary outage
2. Verify certificate identity - Must match stella-ops.org/git.stella-ops.org:ref:refs/tags/v<VERSION>
3. Check OIDC issuer - Must be https://git.stella-ops.org (Gitea), not the GitHub Actions issuer

Rollback Failed

1. Verify target version exists - Check artifact storage
2. Check Helm/kubectl access - Cluster credentials
3. Review health check - Service may need manual intervention
4. Check resource constraints - May need to scale down first

Related Documentation

• README - CI/CD Overview
• Workflow Triggers
• Versioning Guide
• .gitea/README.md (active vs. archived workflow catalog; secrets/variables reference)
• .gitea/workflows-archived/ (source YAML for the pipelines described here)

Container Registry Guide and Helm Deployment Guide are referenced by the legacy CI docs but have not been migrated into this docs tree yet; links omitted to avoid dead references.