Runbook — Live-stack CI lane operations

Applies to: the live-stack CI lanes that stand up the Stella Ops stack from a bare checkout on the isolated apparmor-host runner: e2e-analyzer-coverage.yml (the heaviest), auto-sbom-readiness-e2e.yml, compliance-golden-path.yml and console-layout-audit.yml, all under .gitea/workflows/. The latest verdict of every lane on main, with its first failing log line and owner, is on the dated CI board. These reproduce the whole deploy from clean, so they surface drift no other lane catches. Companion: docs/runbooks/registry/ci-registry-pull-through-cache.md, the live-deploy-operator role playbook (AGENTS.md — Memory), the orchestrator role playbook (AGENTS.md — Memory). Verify against src/ and the workflow YAML — code wins.

Meta-lesson first (it cost ~3 CI rounds): when a CI run’s behavior contradicts what the file shows, run git blame / git diff HEAD before theorizing about YAML/compose internals. Uncommitted working-tree edits to a concurrent file make the runner’s HEAD differ from what you’re reading.

Topology (ruled — Q-9, 2026-08-25: isolated-runner)

Live lanes run against a disposable Stella Ops estate colocated with the isolated runner: the stack is composed up on the runner host per run (or per operational window) and torn down after. The runner is never granted network reach into the remote developer estate — a sometimes-reachable remote estate produces greens that verify nothing, and a standing CI credential into a long-lived estate widens the trust boundary for no gain. Secret VALUES (the STELLAOPS_ADMIN_PASS-class preflights) stay operator-managed on the runner and are never committed. A lane failing because the runner host lacks the estate bring-up or its secrets is an OPERATIONAL provisioning gap (owner-attended), not a product or lane defect. Settlement receipt: docs-archive/implplan/SPRINT_20260825_001_DevOps_ci_gate_liveness_restoration.md Decisions & Risks.

What these lanes need from a bare checkout (gitignored dev assets)

A fresh runner has none of the operator’s local secrets/certs/env. Provision in this order — each gap only surfaces after the previous is fixed:

1. .env(devops/compose/.env, gitignored) — image pins + ~15 generated secrets + the postgres password embedded literally in the STELLAOPS_POSTGRES_* connection strings. Materialize with devops/compose/scripts/init-local-env.sh --force (renders env/stellaops.env.local-dev.example). Do NOT pin POSTGRES_PASSWORD in workflow env — it desyncs from the connection-string literal. Symptom: invalid empty volume spec / STELLAOPS_POSTGRES_<SERVICE>_CONNECTION must be set for ….
2. Dev TLS pfx — devops/etc/authority/keys/kestrel-dev.pfx is gitignored (only the .crt is committed). Generate with devops/compose/scripts/ensure-dev-certs.sh (self-signed; SAN stella-ops.local,*.stella-ops.local,127.1.0.1,127.0.0.1). Symptom: Authority FileNotFoundException: …/kestrel-dev.pfx.
3. Cert and key perms — generated files are root-owned 0600, but the Authority container runs non-root (UID 10001) over a read-only mount, so it fails with UnauthorizedAccessException. Grant access by ownership: chown -R 10001:10001, or setfacl for UID 10001 when not root. Keep the modes at 0700/0600. The live lanes fail the step if any generated secret is world-readable, so never widen the mode.
4. Scanner surface — append devops/compose/docker-compose.scanner-surface-fix.override.yml (named volume + chown init): scanner surface validation rejects the dev HTTP S3 endpoint, and uid 10001 can’t write the surface tmpfs at mode:1777 on apparmor-host.
5. Vault — registering an SCM integration stores its credential via Vault; wire VAULT_ADDR/VAULT_TOKEN (dev root token) in the e2e override or integrations-web 500s.
6. Authority operator keys — generate devops/etc/secrets/authority-signing.pem and authority-ack-token.pem per run (EC P-256), then grant access as in item 3. Nothing else creates them, and Authority fails closed without them. Symptom: Authority notification ack-token signing key 'dev-ack-key-2' was not found at '/app/etc/secrets/authority-ack-token.pem'.
7. Runtime plugin bundles — .runtime/plugins/* is git-ignored packaged output. Every router-backed service bind-mounts router/base and messaging/base, and integrations-web also mounts integrations/recommended. Package them with devops/build/package-runtime-plugins.ps1 after enrolling the modules’ purpose keys with the Scanner signing tool. Symptom: Transport plugin 'messaging' is not available ... Provide a plugin assembly in '/app/plugins/router/base'. The Scanner pair’s scanner/base analyzer bundles come only from build-service-publish.sh, never from build-all.sh. console-layout-audit, e2e-analyzer-coverage and auto-sbom-readiness-e2e carry the working step.
8. One database per service — since PCD-8 (2026-09-26) a lane whose postgres mounts postgres-init and starts from an EMPTY volume gets every per-service database and owner role from 20-service-databases.sh: docker-compose.stella-infra.yml projects the connection variables, and the local-dev template gives each its own role and generated password. The provisioner runs only against a new data directory. A lane that can reuse a volume, or whose overlay removes the init bind (compliance-golden-path), still provisions by hand after postgres accepts TCP on two consecutive polls (its temporary init server answers only the socket): run tools/scripts/deploy/postgres/provision-service-database.sh <database> <role> for every STELLAOPS_POSTGRES_<SERVICE>_CONNECTION the started services receive. It is idempotent on a provisioned volume. Pass the password through PGPASSWORD_SERVICE and read rows from FD 3, because the script drains stdin. Symptom: 3D000: database "stellaops_<service>" does not exist. auto-sbom-readiness-e2e, e2e-analyzer-coverage and compliance-golden-path list their rows; compliance-golden-path provisions every connection the env preparer writes, because the isolation probe walks all of them.
9. router.stella-ops.local for Authority over TLS — Authority serves HTTP only, and clients fetch its token and OIDC metadata at https://router.stella-ops.local/. A lane whose services call Authority must start router-gateway: build it; override it with depends_on: !reset {} and ports: !reset [], because the runner’s nginx owns 80/443; copy the signing CA into trust-roots/router-plugins; and wait until the metadata route answers and 443 is listening. Symptom: Name or service not known (router.stella-ops.local:443). A lane that queries a service’s tables directly must target that service’s own database, such as stellaops_integrations, never stellaops_platform.

Compliance golden-path disposable estate (CIL-2)

.gitea/workflows/compliance-golden-path.yml follows the same Q-9 topology but deliberately does not boot the full stack. It builds and retains one exact-SHA cohort in three serialized jobs, to stay well inside the host runner’s per-job ceiling (timeout: 160m in its config, read on 2026-09-25; it does not enforce a job’s timeout-minutes). Then the fourth job creates only the services used by the signed compliance journey. No image is pushed.

The runtime chain is exactly, in order:

1. devops/compose/docker-compose.stella-ops.yml (one merged model containing infrastructure, services and Evidence)
2. devops/compose/docker-compose.integrations.yml (customer-registry simulator only)
3. devops/compose/docker-compose.compliance-golden-path.override.yml (last)

The canonical model defines both Evidence replica roles. The compliance override replaces their names, image tags, key material and resources for the disposable estate. Do not append the Evidence family file a second time. The lane builds and starts evidence-web only; evidence-worker appears in retained topology but the journey exercises no worker surface.

prepare-compliance-ci-env.sh creates the ignored mode-0600 .env, including distinct Authority client secrets and the service-owned database connections that the full render requires. It direction-checks every ${VAR:?} in the same file list, in the same order, so a newly required input makes the helper red instead of surfacing as a late Compose interpolation failure.

The disposable PostgreSQL override retains only its per-run data volume and omits the inherited predecessor initialization scripts. Those scripts pre-create Authority schemas as the cluster administrator, which prevents the constrained service owner from creating its migration ledger. Each service creates its own schemas through startup migrations on an empty database. The workflow contract renders the exact Compose chain and proves that restoring the initialization bind fails the guard.

PostgreSQL starts first; the workflow provisions and isolation-probes authority, evidence, integrations, jobengine, notify, policy, registry, sbomservice, and scanner before any application starts. Platform is outside these nine provisioned service databases. The required NOLOGIN migration admin roles are pre-created separately; each service runtime retains its constrained role. JobEngine is defined by the canonical model but is not started by this bounded application cohort. Its generated owner connection and Doctor-client secret satisfy the full render and Authority bootstrap contracts.

Authority also supplies its reviewed migration tenant contexts through its owner-specific execution policy. Default-only operational repairs use their declared tenant; estate-wide permission retirements traverse actual catalog tenants within the existing transaction. A role bypass or a blanket default session is not a substitute for this policy. Immutable SQL and ledger checksums remain unchanged.

The overlay changes every selected fixed container name and every selected physical volume to the stellaops-compliance-* namespace, uses stellaops-compliance-network and stellaops-compliance-frontdoor, and remaps only the two host-facing fixtures:

• Gateway: 127.1.0.1:18080 (HTTP) and 127.1.0.1:18443 (HTTPS).
• Customer registry: 127.1.2.9:15000.

The journey therefore sets BASE_URL=https://stella-ops.local:18443, RESOLVE_PORT=18443, REGISTRY_HOST=127.1.2.9:15000, and PG_CONTAINER=stellaops-compliance-postgres. It extracts only the generated admin password key from .env, masks it before use, and never shell-sources the file (PostgreSQL DSNs contain semicolons). The per-run ExportCenter signing key’s public SPKI and fingerprint replace the stale development trust root in the disposable container only.

Three private keys are generated per run into the git-ignored devops/etc/secrets/, because a fresh checkout carries none and each host fails closed without its own: authority-signing, authority-ack-token, and attestor-signing. The third is the consolidated Evidence roles’ local transparency-log checkpoint signing key, and it differs from the other two in one way that matters: its public half IS pinned, inline and in two shapes, by docker-compose.evidence.yml. The lane therefore derives the public half and feeds it back through the override, exactly as it does for the SoA trust root. Mounting a private half that does not match the inline pin renders perfectly and then fails at verification time.

The messaging and Notify runtime-plugin keys are resolved by purpose through Stella Ops cryptography before packaging. In a fresh disposable checkout, CI explicitly enrolls each missing module key with the existing signing tool’s enroll-runtime-key command, then exports its public trust. Existing usable keys are reused; enrollment refuses to replace an existing unusable key. The packager receives the same ignored .local-secrets/compliance-runtime-signing root and continues to fail closed if lookup or signing fails. Private material is neither uploaded with evidence nor copied from the long-lived developer estate.

Authority validates the complete Standard bootstrap-client descriptor, including clients whose services are outside this bounded cohort. The environment preparer supplies the Signals and OfflineKit Doctor client credentials and the CI override forwards them to Authority. The rendered-topology test derives every environment-backed confidential client from standard.yaml and requires a nonempty container value, so adding a descriptor without provisioning fails before the disposable estate starts.

All host-stack workflows share concurrency group host-stack-e2e. The compliance runtime job pre-cleans only its own project, refuses any other stellaops-* fixed-name estate or occupied high port, uploads journey/build/topology/database evidence on red or green, and then always runs project-scoped down -v --remove-orphans. Completion additionally proves no project container, run-scoped network, run-scoped volume, or exact-SHA Scanner cache remains. A health check, image build, or successful teardown without every build/journey artifact is not a compliance pass. This lane therefore uses the Gitea-compatible upload-artifact@v3 action as a hard evidence gate rather than applying the general continue-on-error artifact-plumbing allowance below.

The recurring failure classes (durable — these recur across lanes and real deploys)

• OIDC metadata over HTTPS to a self-signed Authority → token validation fails. .NET-on-Linux won’t reliably trust the self-signed cert. Fix/posture: every resource server fetches OIDC metadata over the internal http://authority.stella-ops.local/.well-known/openid-configuration(RequireHttpsMetadata already false; the public issuer stays https). This is the consistent dev posture — see docs/runbooks/authority/authority-gateway-auth-troubleshooting.md §1.
• Cached :dev image drift — the big one. e2e lanes that run cached images are blind to any change in a service’s source or mounted-config contract. A service can crash-loop on an old message (e.g. Authority bootstrap-secret handling) while current source is correct. Fix: add the changed service to the lane’s ALWAYS_REBUILD (the analyzer lane already rebuilds scanner/authority/integrations each run; airgap rebuilds concelier/notify/authority). The pragmatic systemic guard = rebuild every service whose source/mounted-config changed. This drift is main-wide — the real deployed stack needs the same rebuilds.
• Multi-tenant client_credentials → ambiguous tenant. A client with tenants: "a b" and no default can’t resolve a tenant on a client_credentials grant (no account to default from — unlike a password grant). Fix: set the client’s DefaultTenant (e.g. CONCELIER_AUTHORITY__DEFAULTTENANT), don’t omit the tenant. Symptom: outbound auth 400 invalid_request, often swallowed (best-effort publish) so a downstream notification silently never arrives.
• Password grant tenant selection. A password grant authenticates an account; the tenant is the account’s home tenant, never a required request param (requireExplicitSelection=false for password grant). An explicit tenant is still validated against the client allow-list.
• A new required Compose variable breaks every live lane at once. Adding ${VAR:?} to a canonical Compose file without a generated value makes every lane that renders .env from the template stop at interpolation, before any service starts: error while interpolating ... required variable VAR is missing a value, or prepare-compliance-ci-env: required values still missing: VAR. On 2026-09-24 one such variable, JOBENGINE_ANCHOR_AUTHORITY_CLIENT_SECRET, stopped console-layout-audit, compliance-golden-path, e2e-analyzer-coverage and auto-sbom-readiness-e2e. Fix: in the same change, add a __GENERATE_<VAR>__ marker to devops/compose/env/stellaops.env.local-dev.example, its replace_marker line to init-local-env.sh, and the name to prepare-compliance-ci-env.sh when the value is a client secret. devops/compose/scripts/tests/env-template-covers-required-vars.test.py catches the omission, and compliance-script-tests runs it on every push.
• Ephemeral lane images and the SBOM gate. devops/docker/build-all.sh fails in CI when syft is absent, so release builds cannot skip SBOM generation silently. A lane that builds estate-only images it never publishes sets STELLA_SKIP_SBOM=1 on that call. Without it the lane builds every image and then stops on [build-all] FATAL: syft not found and CI is set.
• Port 80 on the shared runner. A container on the apparmor-host runner holds 0.0.0.0:80, so any 127.x:80 publish that a lane starts fails at compose up with Bind for 0.0.0.0:80 failed: port is already allocated. Fix: in the lane’s last-layered override, give every service the lane starts a ports: !override list that republishes only the port the lane uses, on a high host port. Also count the services that depends_on brings in. The analyzer, auto-SBOM and console-audit overrides all do this for Authority.
• A host-mode job goes silent and is killed hours later, with no error line. The apparmor-host-runner (act_runner v0.6.1, /opt/stella-host-runner/config.yaml: capacity: 2, timeout: 160m, on an 11 GB host) did not enforce a job’s timeout-minutes in auto-sbom run 21659. A step that stalled at 13:42 produced no output until the runner’s own 160-minute limit ended the job at about 16:03. Fix: give each long build an in-step budget, run it in its own process group, and capture its output to a file under the evidence directory. At the deadline, list the process group before killing it, so the stall names the process that hung. Close stdin for every pwsh call. auto-sbom-readiness-e2e’s run_bounded is the working shape. Enforced floor: every run: step of an apparmor-host job executes under the job default shell timeout --kill-after=60 <N>m bash --noprofile --norc -e -o pipefail {0}, where N is at most the job’s timeout-minutes, so a stall ends with exit 124 at that limit. tools/scripts/validate/check-host-runner-step-bounds.py (repo-hygiene lane) fails on a host job without it, on a step shell: that escapes it, and on a budget above the job limit. A new host job needs the defaults.run.shell block, and its steps must not set shell: bash. The test-manifest sweeps also end any suite after 45 minutes (run-test-plan.py --suite-timeout-minutes) and write the hung process group to that suite’s log first. The old host-runner-timeout-probe.yml was deleted on 2026-10-28 (C-02 step 1); to prove the bound after a runner upgrade, read the exit 124 of any bounded host-lane step that hit its limit.
• A lane-built image crash-loops on FileNotFoundException for a *.Plugin.* assembly. In mounted plugin mode, devops/docker/build-all.sh and build-service-publish.sh both prune every *.Plugin.*.dll from a service’s app context, except the host-runtime assemblies each one’s is_host_runtime_plugin_payload names. If a host dependency is missing from one list, images from that builder fail at startup while images from the other run. The build log shows [mounted-plugin-prune] <service> removing N plugin payload file(s). Fix: add the assembly to both lists in the same change. devops/compose/scripts/tests/host-runtime-plugin-allowlist-parity.test.py fails when they differ.
• A container-runner step reports bullseye-security expiry or package 404. This was the historical default-image failure: the ubuntu-latest scheduling alias selected Node 20 Bullseye. PCD-28 remapped the four default aliases on 2026-10-05 to the published CI-only Node 22/Python 3.9 Bookworm image by measured repository digest, after normal APT/bootstrap and source qualification. Its PyYAML wheel belongs to the default Python 3.9 interpreter. Seven temporary job overrides are removed; workflow-specific Node setup steps still select their declared version. Inspect the actual job’s Start image line and runtime mapping before diagnosing package failures; the alias alone does not identify the OS. Operator procedure: runner configuration and scoped pull credentials. A host CLI prefetch can pass while the daemon’s pull fails before checkout because its Docker context has no registry reader credential. Qualify the daemon context with a fresh image creation, reader upload denial and publisher upload success. Keep APT signature and expiry verification enabled during replacement qualification.
• A harness that duplicates a service block misses a key the host newly requires at startup. When a service starts failing closed without a key, the change adds the key to the deployment overlays, but a rig that copies the service block does not get it. The container then exits before the lane’s subject runs. tools/conformance/harness/docker-compose.scratch.yml hit this after OSM-7, with registry-web exiting 139. Fix: add a scratch value to the rig in the same change. Harnesses should print service logs before an EXIT-trap teardown, so the startup exception survives in the job log.

Concurrency

The analyzer lane uses fixed container names (stellaops-valkey, …), so no other disposable estate may overlap it. The former Concelier air-gap lane no longer starts containers after VULN-G6.

Diagnostics that pay off

• Distinguish “secret absent from container” vs “present but app reads empty”: a non-leaking docker inspect <ctr> --format '{{range .Config.Env}}…{{end}}' | awk 'KEY=len:N' dump (lengths only, no values) on the failure path. This instantly localizes bootstrap-secret problems.
• Dump base-service logs on any failure (authority/platform/router-gateway/postgres), not just the feature service — a dependency crash-loop otherwise shows only as a compose “Restarting” line.
• The bring-up step often health-gates only postgres+authority; other services can crash-loop unnoticed — check all container health, not just the gate.

Artifacts caveat

The Gitea runner reports as GHES: actions/upload-artifact@v4 hard-fails the job at upload even when tests passed. Use @v3 with continue-on-error so artifact plumbing never sets the verdict.

Verify (live forcing-function)

A lane is green only when it reaches and passes its real harness (analyzer corpus / the full federation export→download→verify→import→notifier→Mailpit flow), not when the stack merely boots.

Compose health inheritance validation

Run bash tools/scripts/validate/check-compose-healthchecks.sh --self-test and the same command without --self-test after changing a registered Compose chain. The guard reads the RAR, SCN and overlay owner manifests, including the SCN current record. Current chains must retain the accepted predecessor in order and name a digest-pinned current profile. Service identity and source provenance remain the respective owner checker’s responsibility.

An explicitly registered private chain input may use its tracked .yml.example sibling in a clean checkout. This admission covers only a single owned service with image, pull_policy: never and named scalar environment settings. A present private input must have the same shape. The guard rejects untracked examples, redirected paths, missing tracked inputs, additional fields, private includes and healthcheck overrides. An earlier base must still supply an enabled healthcheck that can fail. This does not exempt arbitrary examples or change the shrink-only unchecked-service limits.

For the RAR-owned Platform and Scanner callers, run node tools/scripts/validate/check-local-runtime-sources.cjs --root <committed-checkout> and --self-test. Its EVD-5 tail profiles retain exact accepted image, source and URL values from the September 12 consumer receipt; it checks their tracked examples and any present private inputs. Earlier profile and Gateway pins remain independently checked. The Platform Notify queue check compares the canonical JobEngine Scheduler producer; a retired scheduler-web declaration cannot satisfy that producer requirement.

The SCN source checker (node tools/scripts/validate/check-local-scn-runtime-sources.cjs) accepts only the three registered EVD-5 Policy/ReleaseOrchestrator .example profiles when their private files are absent. Their committed examples must have the exact accepted owner, image and environment values. Present private files are preferred and checked too; missing, untracked, redirected or altered examples fail. Its receipt lists substituted profiles and keeps executionReady: false. Runtime replay/probe/preflight operations still require actual inputs and cannot substitute examples. Run the checker’s --self-test for source admission and unchanged current identity rejection controls.

CI compose volume names

A lane that brings up a compose stack sets a unique COMPOSE_PROJECT_NAME and names every named volume from it. Its teardown stays docker compose down -v. The base files pin host-global names such as compose_postgres-data and stellaops-vault-data, and a standing estate on the same host uses the same names. A unique project name alone does not protect them: down -v removes every named volume the merged file set declares.

Each lane override therefore renames every named volume in its closure, as ${COMPOSE_PROJECT_NAME:?...}-<key> or from a lane prefix variable with the same value. Unnamed volumes are already project-scoped. The guard below fails when a base file adds a named volume that a lane override does not rename, when a lane runs as project stellaops or compose, or when a workflow line names an estate volume directly. Estate names are read from the literal volume name: values in devops/compose/**.

bash tools/scripts/validate/check-ci-compose-volume-names.sh --self-test
bash tools/scripts/validate/check-ci-compose-volume-names.sh

The compose healthcheck lane runs both. To fix a finding, add the named key to the lane override the finding names. To prove the result, render the lane with its own project and check that every volume name starts with the project.