Runbook — Live-stack CI lane operations (analyzer-coverage, airgap-federation-e2e)

Applies to: the heavy CI lanes that stand up the full ~40-service stack from a bare checkout (.gitea/workflows/e2e-analyzer-coverage.yml, airgap-federation-e2e.yml; runs-on: apparmor-host). These reproduce the whole deploy from clean, so they surface drift no other lane catches. Companion: docs/runbooks/ci-registry-pull-through-cache.md, docs/agent-playbooks/live-deploy-operator.md, docs/agent-playbooks/orchestrator.md. Verify against src/ and the workflow YAML — code wins.

Meta-lesson first (it cost ~3 CI rounds): when a CI run’s behavior contradicts what the file shows, run git blame / git diff HEAD before theorizing about YAML/compose internals. Uncommitted working-tree edits to a concurrent file make the runner’s HEAD differ from what you’re reading.

What these lanes need from a bare checkout (gitignored dev assets)

A fresh runner has none of the operator’s local secrets/certs/env. Provision in this order — each gap only surfaces after the previous is fixed:

.env(devops/compose/.env, gitignored) — image pins + ~15 generated secrets + the postgres password embedded literally in the STELLAOPS_POSTGRES_* connection strings. Materialize with devops/compose/scripts/init-local-env.sh --force (renders env/stellaops.env.local-dev.example). Do NOT pin POSTGRES_PASSWORD in workflow env — it desyncs from the connection-string literal. Symptom: invalid empty volume spec / STELLAOPS_POSTGRES_CONNECTION not set.
Dev TLS pfx — devops/etc/authority/keys/kestrel-dev.pfx is gitignored (only the .crt is committed). Generate with devops/compose/scripts/ensure-dev-certs.sh (self-signed; SAN stella-ops.local,*.stella-ops.local,127.1.0.1,127.0.0.1). Symptom: Authority FileNotFoundException: …/kestrel-dev.pfx.
Cert perms — generated files are root-owned 0600; the Authority container runs non-root (UID 10001) over a read-only mount → UnauthorizedAccessException. chmod 0644 the key/crt/pfx.
Scanner surface — append devops/compose/docker-compose.scanner-surface-fix.override.yml (named volume + chown init): scanner surface validation rejects the dev HTTP S3 endpoint, and uid 10001 can’t write the surface tmpfs at mode:1777 on apparmor-host.
Vault — registering an SCM integration stores its credential via Vault; wire VAULT_ADDR/VAULT_TOKEN (dev root token) in the e2e override or integrations-web 500s.

The recurring failure classes (durable — these recur across lanes and real deploys)

OIDC metadata over HTTPS to a self-signed Authority → token validation fails. .NET-on-Linux won’t reliably trust the self-signed cert. Fix/posture: every resource server fetches OIDC metadata over the internal http://authority.stella-ops.local/.well-known/openid-configuration(RequireHttpsMetadata already false; the public issuer stays https). This is the consistent dev posture — see docs/runbooks/authority-gateway-auth-troubleshooting.md §1.
Cached :dev image drift — the big one. e2e lanes that run cached images are blind to any change in a service’s source or mounted-config contract. A service can crash-loop on an old message (e.g. Authority bootstrap-secret handling) while current source is correct. Fix: add the changed service to the lane’s ALWAYS_REBUILD (the analyzer lane already rebuilds scanner/authority/integrations each run; airgap rebuilds concelier/notify/authority). The pragmatic systemic guard = rebuild every service whose source/mounted-config changed. This drift is main-wide — the real deployed stack needs the same rebuilds.
Multi-tenant client_credentials → ambiguous tenant. A client with tenants: "a b" and no default can’t resolve a tenant on a client_credentials grant (no account to default from — unlike a password grant). Fix: set the client’s DefaultTenant (e.g. CONCELIER_AUTHORITY__DEFAULTTENANT), don’t omit the tenant. Symptom: outbound auth 400 invalid_request, often swallowed (best-effort publish) so a downstream notification silently never arrives.
Password grant tenant selection. A password grant authenticates an account; the tenant is the account’s home tenant, never a required request param (requireExplicitSelection=false for password grant). An explicit tenant is still validated against the client allow-list.

Concurrency

Base-stack lanes use fixed container names (stellaops-valkey, …), so analyzer-coverage and airgap-federation-e2e cannot run concurrently (container name already in use). Keep their crons de-staggered or project-scope the names; add a CI concurrency group.

Diagnostics that pay off

Distinguish “secret absent from container” vs “present but app reads empty”: a non-leaking docker inspect <ctr> --format '{{range .Config.Env}}…{{end}}' | awk 'KEY=len:N' dump (lengths only, no values) on the failure path. This instantly localizes bootstrap-secret problems.
Dump base-service logs on any failure (authority/platform/router-gateway/postgres), not just the feature service — a dependency crash-loop otherwise shows only as a compose “Restarting” line.
The bring-up step often health-gates only postgres+authority; other services can crash-loop unnoticed — check all container health, not just the gate.

Artifacts caveat

The Gitea runner reports as GHES: actions/upload-artifact@v4 hard-fails the job at upload even when tests passed. Use @v3 with continue-on-error so artifact plumbing never sets the verdict.

Verify (live forcing-function)

A lane is green only when it reaches and passes its real harness (analyzer corpus / the full federation export→download→verify→import→notifier→Mailpit flow), not when the stack merely boots.