Runbook — Live-stack CI lane operations

Applies to: the heavy analyzer CI lane that stands up the full ~40-service stack from a bare checkout (.gitea/workflows/e2e-analyzer-coverage.yml; runs-on: apparmor-host). These reproduce the whole deploy from clean, so they surface drift no other lane catches. Companion: docs/runbooks/registry/ci-registry-pull-through-cache.md, the live-deploy-operator role playbook (AGENTS.md — Memory), the orchestrator role playbook (AGENTS.md — Memory). Verify against src/ and the workflow YAML — code wins.

Meta-lesson first (it cost ~3 CI rounds): when a CI run’s behavior contradicts what the file shows, run git blame / git diff HEAD before theorizing about YAML/compose internals. Uncommitted working-tree edits to a concurrent file make the runner’s HEAD differ from what you’re reading.

Topology (ruled — Q-9, 2026-08-25: isolated-runner)

Live lanes run against a disposable Stella Ops estate colocated with the isolated runner: the stack is composed up on the runner host per run (or per operational window) and torn down after. The runner is never granted network reach into the remote developer estate — a sometimes-reachable remote estate produces greens that verify nothing, and a standing CI credential into a long-lived estate widens the trust boundary for no gain. Secret VALUES (the STELLAOPS_ADMIN_PASS-class preflights) stay operator-managed on the runner and are never committed. A lane failing because the runner host lacks the estate bring-up or its secrets is an OPERATIONAL provisioning gap (owner-attended), not a product or lane defect. Settlement receipt: docs/implplan/SPRINT_20260825_001_DevOps_ci_gate_liveness_restoration.md Decisions & Risks.

What these lanes need from a bare checkout (gitignored dev assets)

A fresh runner has none of the operator’s local secrets/certs/env. Provision in this order — each gap only surfaces after the previous is fixed:

  1. .env(devops/compose/.env, gitignored) — image pins + ~15 generated secrets + the postgres password embedded literally in the STELLAOPS_POSTGRES_* connection strings. Materialize with devops/compose/scripts/init-local-env.sh --force (renders env/stellaops.env.local-dev.example). Do NOT pin POSTGRES_PASSWORD in workflow env — it desyncs from the connection-string literal. Symptom: invalid empty volume spec / STELLAOPS_POSTGRES_<SERVICE>_CONNECTION must be set for ….
  2. Dev TLS pfx — devops/etc/authority/keys/kestrel-dev.pfx is gitignored (only the .crt is committed). Generate with devops/compose/scripts/ensure-dev-certs.sh (self-signed; SAN stella-ops.local,*.stella-ops.local,127.1.0.1,127.0.0.1). Symptom: Authority FileNotFoundException: …/kestrel-dev.pfx.
  3. Cert perms — generated files are root-owned 0600; the Authority container runs non-root (UID 10001) over a read-only mount → UnauthorizedAccessException. chmod 0644 the key/crt/pfx.
  4. Scanner surface — append devops/compose/docker-compose.scanner-surface-fix.override.yml (named volume + chown init): scanner surface validation rejects the dev HTTP S3 endpoint, and uid 10001 can’t write the surface tmpfs at mode:1777 on apparmor-host.
  5. Vault — registering an SCM integration stores its credential via Vault; wire VAULT_ADDR/VAULT_TOKEN (dev root token) in the e2e override or integrations-web 500s.

Compliance golden-path disposable estate (CIL-2)

.gitea/workflows/compliance-golden-path.yml follows the same Q-9 topology but deliberately does not boot the full stack. It builds and retains one exact-SHA cohort in three serialized jobs (the host runner enforces a 60-minute per-job ceiling), then the fourth job creates only the services used by the signed compliance journey. No image is pushed.

The runtime chain is exactly, in order:

  1. devops/compose/docker-compose.stella-ops.yml (one merged model containing infrastructure, services and Evidence)
  2. devops/compose/docker-compose.integrations.yml (customer-registry simulator only)
  3. devops/compose/docker-compose.compliance-golden-path.override.yml (last)

The canonical model defines both Evidence replica roles. The compliance override replaces their names, image tags, key material and resources for the disposable estate. Do not append the Evidence family file a second time. The lane builds and starts evidence-web only; evidence-worker appears in retained topology but the journey exercises no worker surface.

prepare-compliance-ci-env.sh creates the ignored mode-0600 .env, including distinct Authority client secrets and the service-owned database connections that the full render requires. It direction-checks every ${VAR:?} in the same file list, in the same order, so a newly required input makes the helper red instead of surfacing as a late Compose interpolation failure.

The disposable PostgreSQL override retains only its per-run data volume and omits the inherited predecessor initialization scripts. Those scripts pre-create Authority schemas as the cluster administrator, which prevents the constrained service owner from creating its migration ledger. Each service creates its own schemas through startup migrations on an empty database. The workflow contract renders the exact Compose chain and proves that restoring the initialization bind fails the guard.

PostgreSQL starts first; the workflow provisions and isolation-probes authority, evidence, integrations, jobengine, notify, policy, registry, sbomservice, and scanner before any application starts. Platform is outside these nine provisioned service databases. The required NOLOGIN migration admin roles are pre-created separately; each service runtime retains its constrained role. JobEngine is defined by the canonical model but is not started by this bounded application cohort. Its generated owner connection and Doctor-client secret satisfy the full render and Authority bootstrap contracts.

Authority also supplies its reviewed migration tenant contexts through its owner-specific execution policy. Default-only operational repairs use their declared tenant; estate-wide permission retirements traverse actual catalog tenants within the existing transaction. A role bypass or a blanket default session is not a substitute for this policy. Immutable SQL and ledger checksums remain unchanged.

The overlay changes every selected fixed container name and every selected physical volume to the stellaops-compliance-* namespace, uses stellaops-compliance-network and stellaops-compliance-frontdoor, and remaps only the two host-facing fixtures:

The journey therefore sets BASE_URL=https://stella-ops.local:18443, RESOLVE_PORT=18443, REGISTRY_HOST=127.1.2.9:15000, and PG_CONTAINER=stellaops-compliance-postgres. It extracts only the generated admin password key from .env, masks it before use, and never shell-sources the file (PostgreSQL DSNs contain semicolons). The per-run ExportCenter signing key’s public SPKI and fingerprint replace the stale development trust root in the disposable container only.

Three private keys are generated per run into the git-ignored devops/etc/secrets/, because a fresh checkout carries none and each host fails closed without its own: authority-signing, authority-ack-token, and attestor-signing. The third is the consolidated Evidence roles’ local transparency-log checkpoint signing key, and it differs from the other two in one way that matters: its public half IS pinned, inline and in two shapes, by docker-compose.evidence.yml. The lane therefore derives the public half and feeds it back through the override, exactly as it does for the SoA trust root. Mounting a private half that does not match the inline pin renders perfectly and then fails at verification time.

The messaging and Notify runtime-plugin keys are resolved by purpose through Stella Ops cryptography before packaging. In a fresh disposable checkout, CI explicitly enrolls each missing module key with the existing signing tool’s enroll-runtime-key command, then exports its public trust. Existing usable keys are reused; enrollment refuses to replace an existing unusable key. The packager receives the same ignored .local-secrets/compliance-runtime-signing root and continues to fail closed if lookup or signing fails. Private material is neither uploaded with evidence nor copied from the long-lived developer estate.

Authority validates the complete Standard bootstrap-client descriptor, including clients whose services are outside this bounded cohort. The environment preparer supplies the Signals and OfflineKit Doctor client credentials and the CI override forwards them to Authority. The rendered-topology test derives every environment-backed confidential client from standard.yaml and requires a nonempty container value, so adding a descriptor without provisioning fails before the disposable estate starts.

All host-stack workflows share concurrency group host-stack-e2e. The compliance runtime job pre-cleans only its own project, refuses any other stellaops-* fixed-name estate or occupied high port, uploads journey/build/topology/database evidence on red or green, and then always runs project-scoped down -v --remove-orphans. Completion additionally proves no project container, run-scoped network, run-scoped volume, or exact-SHA Scanner cache remains. A health check, image build, or successful teardown without every build/journey artifact is not a compliance pass. This lane therefore uses the Gitea-compatible upload-artifact@v3 action as a hard evidence gate rather than applying the general continue-on-error artifact-plumbing allowance below.

The recurring failure classes (durable — these recur across lanes and real deploys)

Concurrency

The analyzer lane uses fixed container names (stellaops-valkey, …), so no other disposable estate may overlap it. The former Concelier air-gap lane no longer starts containers after VULN-G6.

Diagnostics that pay off

Artifacts caveat

The Gitea runner reports as GHES: actions/upload-artifact@v4 hard-fails the job at upload even when tests passed. Use @v3 with continue-on-error so artifact plumbing never sets the verdict.

Verify (live forcing-function)

A lane is green only when it reaches and passes its real harness (analyzer corpus / the full federation export→download→verify→import→notifier→Mailpit flow), not when the stack merely boots.

Compose health inheritance validation

Run bash tools/scripts/validate/check-compose-healthchecks.sh --self-test and the same command without --self-test after changing a registered Compose chain. The guard reads the RAR, SCN and overlay owner manifests, including the SCN current record. Current chains must retain the accepted predecessor in order and name a digest-pinned current profile. Service identity and source provenance remain the respective owner checker’s responsibility.

An explicitly registered private chain input may use its tracked .yml.example sibling in a clean checkout. This admission covers only a single owned service with image, pull_policy: never and named scalar environment settings. A present private input must have the same shape. The guard rejects untracked examples, redirected paths, missing tracked inputs, additional fields, private includes and healthcheck overrides. An earlier base must still supply an enabled healthcheck that can fail. This does not exempt arbitrary examples or change the shrink-only unchecked-service limits.

For the RAR-owned Platform and Scanner callers, run node tools/scripts/validate/check-local-runtime-sources.cjs --root <committed-checkout> and --self-test. Its EVD-5 tail profiles retain exact accepted image, source and URL values from the September 12 consumer receipt; it checks their tracked examples and any present private inputs. Earlier profile and Gateway pins remain independently checked. The Platform Notify queue check compares the canonical JobEngine Scheduler producer; a retired scheduler-web declaration cannot satisfy that producer requirement.

The SCN source checker (node tools/scripts/validate/check-local-scn-runtime-sources.cjs) accepts only the three registered EVD-5 Policy/ReleaseOrchestrator .example profiles when their private files are absent. Their committed examples must have the exact accepted owner, image and environment values. Present private files are preferred and checked too; missing, untracked, redirected or altered examples fail. Its receipt lists substituted profiles and keeps executionReady: false. Runtime replay/probe/preflight operations still require actual inputs and cannot substitute examples. Run the checker’s --self-test for source admission and unchanged current identity rejection controls.