Release & Operations Architecture — Stella Ops Suite (2025 Q4)

Audience: release engineers, platform operators, and architects.

Draws from the AOC guardrails, Orchestrator, Export Center, and Observability module plans to describe how Stella Ops is built, signed, distributed, and operated.

Scope. Implementation‑ready blueprint for how Stella Ops is built, versioned, signed, distributed, upgraded, licensed (PoE), and operated in customer environments (online and air‑gapped). Covers reproducible builds, supply‑chain attestations, registries, offline kits, migration/rollback, artifact lifecycle (RustFS default + PostgreSQL, S3 fallback), monitoring SLOs, and customer activation.

Deployment scope. Kubernetes and Helm are explicitly unsupported. Stella Ops release operations target Docker Compose, Offline Kit, and signed release manifests for non-Kubernetes container estates.

0) Product vision (operations lens)

Stella Ops must be trustable at a glance and boringly operable:

1) Release trains & versioning

1.1 Channels

Reality check (flagged). The release tooling recognises the channels stable, airgap, and edge(produce_image_digests.py defaults to ("stable", "airgap"); components.json sets defaultChannel: "edge"). There is no lts channel in the tooling today, and airgapis a first-class channel not listed above. The default channel in tooling is edge, whereas this section names stable as the default. Reconcile the channel taxonomy with devops/release/ before relying on the list above.

1.2 Version strings

Semantic core + calendar tag:

<MAJOR>.<MINOR>.<PATCH>  (<YYYY>.<MM>)   e.g., 2.4.1 (2027.06)

Divergence (flagged). The semver-plus-calendar scheme above is the stated policy, but the release manifests actually checked into devops/releases/ use a calendar-versioned version string directly — e.g. version: "2025.09.2" with calendar: "2025.09" (and 2025.09.2-airgap for the air-gap channel). There is no separate 2.4.1-style semver field on disk. Reconcile this section with the product owner before treating the MAJOR.MINOR.PATCH form as authoritative.

1.3 Component alignment

A release is a bundle of image digests, Compose assets, offline-kit metadata, and manifests. All services in a bundle are wire‑compatible. Mixed minor versions are allowed within a bounded skew:

At startup, services self‑advertise their semver & channel; the UI surfaces mismatch warnings.

2) Supply‑chain pipeline (how a release is built)

2.1 Deterministic builds

2.2 First‑party SBOMs & provenance

2.3 Signing & transparency

2.4 Gates & tests

2.5 Debug-store artefacts

3) Distribution & activation

3.1 Registries

Gating policy:

Monetization lever is signing (PoE gate), not image pulls, so the core remains simple to consume.

3.2 OAuth2 token service (for private repos)

Flagged (unsubstantiated). Earlier drafts described these registry tokens as DPoP‑bound, but StellaOps.Registry.TokenService shows no DPoP / cnf / jkt proof-of-possession binding — the issued JWT is a short-lived bearer token scoped to repository:…:pull. Either DPoP binding is not yet implemented for registry tokens (roadmap) or it lives elsewhere; verify before presenting DPoP as a current registry-token control.

The token service enforces plan gating via registry-token.yaml (see docs/modules/registry/operations/token-service.md) and exposes Prometheus metrics (registry_token_issued_total, registry_token_rejected_total). Revoked licence identifiers halt issuance even when scope requirements are met.

3.3 Offline kits (air‑gapped)

4) Licensing (PoE) & monetization

Principle: Only paid Stella Ops issues valid signed attestations. Running the stack is free; signing requires PoE.

4.1 PoE issuance

4.2 Online enforcement

4.3 Air‑gapped / offline

4.4 Stolen/abused PoE

5) Deployment path (customer side)

5.1 First install

Stella Ops first install is Compose or Offline Kit based. Kubernetes and Helm are not supported deployment targets.

The Compose entry point is devops/compose/docker-compose.stella-ops.yml, a thin wrapper that include:s docker-compose.stella-infra.yml (Postgres, Valkey, RustFS, zot registry, Rekor) and docker-compose.stella-services.yml (the application services). Generate local environment values, then bring the stack up:

cd devops/compose
./scripts/init-local-env.sh
docker compose --env-file .env -f docker-compose.stella-ops.yml up -d

Note. Earlier drafts of this blueprint referenced devops/compose/env/prod.env and devops/compose/docker-compose.prod.yaml. Neither exists; local dev uses devops/compose/env/stellaops.env.local-dev.example through scripts/init-local-env.*; operator/release installs use devops/compose/env/stellaops.env.example. The canonical Compose file is docker-compose.stella-ops.yml. Overlays such as docker-compose.telemetry.yml, docker-compose.dev-ui.yml, docker-compose.blue-green.yml, and the docker-compose.compliance-*.yml profiles layer on top with additional -f flags.

Offline environments import the signed Offline Kit first, then apply the included Compose profile and release manifest. Post-install registration still creates Authority clients for Scanner, Signer, Attestor, and UI, and the UI banner shows the release bundle and verification state.

5.2 Updates

Reconcile with ADR-004 (flagged). The platform migration policy is now forward-only (see docs/architecture/decisions/ADR-004-forward-only-migrations.md, Accepted 2026-05-18, and the migration-recovery runbook). Under forward-only, a true schema downgrade is not performed by redeploying prior images; downgrades are recovered by PostgreSQL snapshot restore. Services auto-migrate their own schema on startup (CLAUDE.md §2.7), so the “keep both schemas valid and roll back images” model above only holds for the brief expand window — past Phase B (drop) it does not. Treat snapshot restore as the authoritative downgrade path.

5.3 Rollback

6) Release payloads & manifests

6.1 Release manifest (release.yaml)

The released-manifest schema as actually emitted and validated lives under devops/releases/ (e.g. 2025.09-stable.yaml, 2025.09-airgap.yaml). Each manifest carries a components: list (name + digest-pinned image), an infrastructure: block (the postgres, valkey, and rustfs images), and a single checksums.releaseManifestSha256:

release:
  version: "2025.09.2"
  channel: "stable"
  date: "2025-09-20T00:00:00Z"
  calendar: "2025.09"
  components:
    - name: authority
      image: registry.stella-ops.org/stellaops/authority@sha256:b0348bad…
    - name: signer
      image: registry.stella-ops.org/stellaops/signer@sha256:8ad574e6…
    - name: scanner-web
      image: registry.stella-ops.org/stellaops/scanner-web@sha256:14b23448…
    # … attestor, scanner-worker, concelier, excititor, notify-web,
    #    advisory-ai-web, advisory-ai-worker, web-ui …
  infrastructure:
    postgres:
      image: docker.io/library/postgres@sha256:8e97b852…
    valkey:
      image: docker.io/valkey/valkey@sha256:9a2cf7c9…
    rustfs:
      image: registry.stella-ops.org/stellaops/rustfs:2025.09.2
  checksums:
    releaseManifestSha256: dc3c8fe1ab83941c838ccc5a8a5862f7ddfa38c2078e580b5649db26554565b7

PARTIALLY IMPLEMENTED (forward-looking shape). The richer per-component manifest envisioned by this blueprint — embedding sbom: / provenance: OCI referrer URIs, per-component signature: { rekorUUID }, and a top-level compose: / charts: block — is not populated in any of the manifests under devops/releases/ (2025.09-stable, 2025.09-airgap, 2025.10-edge, 2025.09-mock-dev). Those manifests carry only name + digest-pinned image per component. SBOM, provenance, and transparency-log entries are attached to images as OCI referrers (§2.2–2.3) and tracked outside release.yaml. The format is forward-compatible: the Offline Kit builder (devops/offline/kit/build_offline_kit.py) does read and stage per-component sbom / provenance entries and charts / compose collections when present, and the validator (devops/release/check_release_manifest.py) tolerates charts: / compose: section headers. But nothing currently emits them — treat the SBOM/provenance/signature manifest fields above as roadmap, not current contract.

The manifest is intended to be cosign‑signed so UI/CLI can verify a bundle without talking to registries.

Deployment guardrails - The repository keeps channel-aligned release manifests in devops/releases/ and Compose bundles in devops/compose/. Helm overlays and Kubernetes manifests are not active release assets. Release engineering still runs devops/release/check_release_manifest.py, which fails if any required component or infrastructure image is missing from the local release manifest or is still tag-only (not @sha256:-pinned). Required components are authority, signer, attestor, scanner-web, scanner-worker, concelier, excititor, notify-web, advisory-ai-web, advisory-ai-worker, web-ui; required infrastructure is postgres, valkey, rustfs. The digest producer/patcher path (devops/release/produce_image_digests.py, apply_image_digests.py) remains useful for non-Kubernetes packaging because it builds local, digest-pinned release-manifest authority without resolving mutable tags.

Flagged (sample manifests do not pass the validator). The REQUIRED_COMPONENTS list above is transcribed accurately from check_release_manifest.py, but the sample manifests checked into devops/releases/ would fail that validator today: 2025.09-stable.yaml and 2025.09-airgap.yaml omit notify-webfrom their components: lists entirely, and they pin advisory-ai-web / advisory-ai-worker by tag (:2025.09.2, :2025.09.2-airgap) rather than @sha256:. Both conditions are hard failures in check_manifest() (“missing required component” / “not digest-pinned”). 2025.10-edge.yaml additionally carries an extra issuer-directory-web component not in the required set (extra components are tolerated). Treat the committed manifests as illustrative fixtures, not validator-clean release authority; a real release run must add notify-web and digest-pin the advisory-ai images. Reconcile with release engineering before citing these files as canonical.

6.2 Image labels (release metadata)

Each image sets OCI labels:

org.opencontainers.image.version = "2.4.1"
org.opencontainers.image.revision = "<git sha>"
org.opencontainers.image.created = "2027-06-20T12:00:00Z"
org.stellaops.release.calendar = "2027.06"
org.stellaops.release.channel  = "stable"
org.stellaops.build.slsaProvenance = "oci://…"

Signer validates scanner image’s cosign identity + calendar tag for release window checks.

7) Artifact lifecycle & storage (RustFS/PostgreSQL)

7.1 Buckets & prefixes (RustFS)

rustfs://stellaops/
  scanner/
    layers/<sha256>/sbom.cdx.json.zst
    images/<imgDigest>/inventory.cdx.pb
    images/<imgDigest>/usage.cdx.pb
    diffs/<old>_<new>/diff.json.zst
    attest/<artifactSha256>.dsse.json
  concelier/
    json/<exportId>/...
    trivy/<exportId>/...
  excititor/
    exports/<exportId>/...
  attestor/
    dsse/<bundleSha256>.json
    proof/<rekorUuid>.json

7.2 ILM classes

7.3 Artifact Lifecycle Controller (ALC)

Migration note. Follow docs/modules/scanner/operations/rustfs-migration.md when transitioning existing MinIO buckets to RustFS. The provided migrator is idempotent and safe to rerun per prefix.

Flagged (ALC / ILM-class scheme is design-level). The named Artifact Lifecycle Controller background worker and the three ILM classes (short / default / compliance) described in §7.2–§7.3 are a blueprint: src/Scanner/__Libraries/StellaOps.Scanner.Storage/ carries artifact-storage rows, object-store adapters (S3ArtifactObjectStore), and EPSS cache retention, but there is no single component named “Artifact Lifecycle Controller” enforcing those exact TTL classes and reference-counting in the tree today. Treat the ILM-class taxonomy and per-class UI purge view as intended design, and verify against StellaOps.Scanner.Storage / Scanner.WebService before presenting them as shipped controls.

7.4 PostgreSQL retention

7.5 PostgreSQL server baseline

8) Observability & SLOs (operations)

Prometheus + OTLP; Grafana dashboards ship in the monitoring bundle.

9) Security & compliance operations

Corrected (MongoDB not fully removed). Earlier drafts asserted “MongoDB dependencies were removed (no MongoDB.Driver references remain under src/); all persistence now uses PostgreSQL.” That is not accurate as written. src/Directory.Packages.props still declares MongoDB.Driver (3.3.0) and MongoDB.Bson, and an active library — src/Workflow/__Libraries/StellaOps.Workflow.DataStore.MongoDB/ (plus its __Tests project) — references MongoDB.Driver and contains live Mongo-backed stores (MongoWorkflowDatabase, projection/runtime/signal stores, etc.). The accurate statement is narrower: the platform/release services (Authority, Scanner, Concelier, Excititor, Attestor, Signer, Notify, Advisory-AI) persist to PostgreSQL (per CLAUDE.md §2.7 auto-migration), and no top-level release service in this blueprint takes a Mongo dependency — but the Workflow datastore is a MongoDB-backed exception still present in the tree. Verify against src/Workflow/ before repeating the blanket “no MongoDB anywhere” claim.

10) Customer update flow (how versions are fetched & activated)

10.1 Online clusters

10.2 Air‑gapped clusters

10.3 CLI self‑update (optional)

11) Compatibility & deprecation policy

12) Runbooks (selected)

For the full promote/upgrade/rollback procedure (digest pinning, manifest validation, migration gating, and the image-staleness self-check), see the Deployment upgrade & rollback runbook.

12.1 Lost PoE

  1. Suspend automatic attestation jobs.
  2. Confirm the entitlement-denied state from the Signer service (the PoE introspector rejects signing requests when the entitlement is revoked/expired/out-of-window). Note: there is no stellaops signer status CLI command — Signer has no dedicated CLI command group; check Signer service health/logs or the attestation UI instead. (Flagged: this runbook step references a CLI verb that does not exist.)
  3. Obtain new PoE from portal; reconfigure the Signer entitlement.
  4. Re‑enable; optionally re‑sign last N reports (UI button → batch).

12.2 Rekor outage (self‑hosted)

12.3 Emergency downgrade

13) Example: cluster bootstrap (Compose)

Illustrative only. This is a hand-written teaching example, not a real file from the repo. The actual stack is assembled from docker-compose.stella-ops.yml (which include:s the infra and services files — see §5.1). The top-level version: key shown below is obsolete in modern Docker Compose (the repo’s active Compose files omit it and Compose now warns when it is present); it is retained here only to keep the example self-contained.

version: "3.9"   # obsolete in current Compose; omit in real files
services:
  authority:
    image: registry.stella-ops.org/stellaops/authority@sha256:...
    env_file: ./env/authority.env
    ports: ["8440:8440"]
  signer:
    image: registry.stella-ops.org/stellaops/signer@sha256:...
    depends_on: [authority]
    environment:
      # Illustrative only. The shipped configured introspector reads the
      # entitlement definition (LicenseId, plan window) from prefixed env vars
      # (see ConfiguredProofOfEntitlementIntrospector); SIGNER__POE__LICENSING__INTROSPECTURL
      # is NOT a recognised key. Consult the Signer module dossier for the
      # current configuration surface.
      - SIGNER__POE__LICENSING__INTROSPECTURL=https://www.stella-ops.org/api/v1/license/introspect
  attestor:
    image: registry.stella-ops.org/stellaops/attestor@sha256:...
    depends_on: [signer]
  scanner-web:
    image: registry.stella-ops.org/stellaops/scanner-web@sha256:...
    environment:
      - SCANNER__ARTIFACTSTORE__ENDPOINT=http://rustfs:8080
  scanner-worker:
    image: registry.stella-ops.org/stellaops/scanner-worker@sha256:...
    deploy: { replicas: 4 }
  concelier:
    image: registry.stella-ops.org/stellaops/concelier@sha256:...
  excititor:
    image: registry.stella-ops.org/stellaops/excititor@sha256:...
  web-ui:
    image: registry.stella-ops.org/stellaops/web-ui@sha256:...
  postgres:
    image: postgres:18.1
  valkey:
    image: valkey/valkey:9.0.1
  rustfs:
    image: registry.stella-ops.org/stellaops/rustfs@sha256:...
  rekor-cli:
    image: ghcr.io/sigstore/rekor-cli:v1.4.3
    profiles: ["sigstore"]
  cosign:
    image: ghcr.io/sigstore/cosign:v3.0.4
    profiles: ["sigstore"]

The cosign (ghcr.io/sigstore/cosign:v3.0.4) and rekor-cli (ghcr.io/sigstore/rekor-cli:v1.4.3) tool containers ship as one-shot helpers in devops/compose/docker-compose.stella-infra.yml under the sigstore profile; enable them with docker compose --profile sigstore. The default Postgres (postgres:18.1), Valkey (valkey/valkey:9.0.1), and RustFS images are supplied via the POSTGRES_IMAGE / VALKEY_IMAGE / RUSTFS_IMAGE env variables (see env/stellaops.env.example) rather than hard-coded in the Compose files, so prod deployments can override with audited digests.

Corrected. There is no devops/compose/docker-compose.rekor-v2.yaml overlay file. A self-hosted Rekor v2 (tiles) server is defined inline as the rekor-v2 service in docker-compose.stella-infra.yml under the sigstore-localprofile (image from REKOR_TILES_IMAGE). It is configured via the rekor-server serve flags (--http-port 3322, --grpc-port 3323, --signer-filepath, --gcp-bucket, --gcp-spanner) — there is no REKOR_SERVER_URL knob on that service. Enable it with docker compose --profile sigstore-local up -d rekor-v2.

14) Governance & keys (who owns the trust root)

15) Roadmap (Ops)

Appendix A — Minimal SLO monitors

Appendix B — Upgrade safety checklist

End — Release & Operations Architecture