Stella Ops Runbooks

Audience: operators running the Stella Ops release control plane (and DevOps vulnerability scanner) in dev or production.

This directory holds operator-facing runbooks for platform operations. Each runbook is self-contained and assumes you have a working install with admin credentials. Start from the Categories index below, or jump to a sibling runbook home if your task is module-specific (see next section).

Which runbook home?

Runbooks live in three intentionally distinct locations. Pick the one that matches your task:

• docs/runbooks/(this directory) — cross-platform operational procedures and drills (KEK rotation, blue/green and canary deploys, backup/restore, determinism tests, tenant onboarding).
• docs/operations/runbooks/ — module-specific incident response (e.g. scanner-oom.md, policy-incident.md, attestor-rekor-unavailable.md, orchestrator-promotion-stuck.md).
• docs/ops/runbooks/ — niche plugin/transport operations (e.g. add-router-transport.md).

Categories

Crypto & Key Management

• verification-key-rotation.md — rotate signature-verification root keys per region profile (packs / VexHub HMAC / signing root) via the ADR-028 phased dual-trust flow: introduce → re-envelope → consequences-gated invalidate; air-gap-safe, fail-closed, with recovery
• kek-rotation.md — rotate the master KEK end-to-end across the three credential-ciphertext stores (platform.connector_credentials, integrations.integrations, release_orchestrator.deployment_bundles)
• credential-storage-algorithm-switch.md — per-tenant credential storage algorithm switch flow for regulated regions (eIDAS / FIPS / GOST R / SM/ShangMi)
• kek-source-vault.md — configure HashiCorp Vault as the master KEK source
• kek-source-hsm.md — configure a PKCS#11 HSM as the master KEK source (hardware-validation caveat applies)
• secret-provider-multi-provider.md — run a mixed secret-provider registry (builtin + external vault/openbao; the in-repo openbao profile is test-only dev/CI, never shipped) via the ADR-032 URL-routed Crypto:SecretProviders[] config; DefaultSealTarget + URL grammar quickstart; fail-closed

Database & Migrations

• migration-recovery.md — recover from a failed forward-only migration via PostgreSQL volume snapshot restore
• backup-restore-drill.md — disaster-recovery drill (DEVOPS-OPS-029-01): back up the postgres state volume, wipe it, and restore cleanly into an empty cluster

Deployment & Upgrades

• deployment-prerequisites.md — operator prerequisites for adding tenants under a single operator trust boundary (anchored in ADR-005)
• ansible-delivery-environment-onboarding.md — onboard an AnsibleHost environment through public APIs, with operator-owned playbooks, task-scoped managed inventory/vars, deployment-scoped secret resolution, and forcing-function verification
• blue-green-deployment.md — compose-level blue/green deployment for one service at a time using docker-compose.blue-green.yml and the switch scripts
• weighted-canary-deployment.md — router-native weighted canary for introducing a new service version to a fraction of live traffic before promote/abort
• bootstrap-determinism-test.md — TOPO-045 bootstrap-from-scratch reproducibility validation with pinned digests, pinned mirror seed, and a baseline diff oracle
• image-rebuild-order.md — canonical rebuild order (authority → scanner-worker → integrations-web) plus the --force-recreate / AGENT_CORE_CAPABILITIES / -NoCache caveats; collects the costly rediscoveries from Sprints 046, 049, 050, 055, 060, 066

Verification & Determinism

• operator-verify.md — Gateway /api/v1/operator/verify/* operator verification surface and the evidence-capture layout for VERIFY-AGG-011
• replay-determinism-test.md — POST /api/v1/tester/replay-and-compare flow to verify two replay runs produce the same canonical server-side artefact hashes

Attestation & Transparency

• attestor-transparency-mirror-exchange.md — mirror local Stella transparency entries to an approved external Rekor-compatible log as additive (non-runtime-dependency) evidence

Registry & Air-Gap

• registry-bridge-cache.md — operate the agent registry bridge pull-through cache: enable the loopback bridge, configure upstreams (CLI + UI) and scoped cache policy, L1/L2 tiers, eviction/pinning, the opt-in central PolicyRefresh, and hit/miss/eviction troubleshooting against the StellaOps.Agent.Registry Grafana board
• ci-registry-pull-through-cache.md — deploy + operate a self-hosted Docker Hub pull-through cache (registry:2 proxy) on the CI runner host; daemon registry-mirror + Testcontainers wiring, air-gap pre-warming, and the broken runner-creds stopgap
• registry-compatibility.md — quick reference for OCI registry compatibility checks via stella doctor --tag registry (see also docs/modules/doctor/registry-checks.md)
• registry-referrer-troubleshooting.md — diagnose and resolve OCI referrer-discovery issues during mirror bundle exports
• golden-corpus-operations.md — golden-corpus infrastructure operations: health checks, validation, incident response, and routine maintenance (GCB-006)

Telemetry & Observability

• federated-telemetry-operations.md — enable/disable federation, manage consent, monitor privacy budgets, and troubleshoot sync failures for the Federated Telemetry subsystem

Runtime & Scanning

• runtime-linkage-ops.md — production deployment and operation of the eBPF-based runtime linkage verification system against declared function maps

Tenancy & Onboarding

• tenant-onboarding.md — onboard a new tenant via POST /console/admin/tenants with first-class auto-grant of CLI client access (Sprint 031): config, opt-out, audit shape, failure modes, manual fallback SQL

CI & Build Lanes

• ci-lane-operations.md — operate the heavy live-stack CI lanes (analyzer-coverage, airgap-federation-e2e) that boot the full stack from a bare checkout: the gitignored-asset provisioning chain (.env / dev certs / scanner-surface / Vault), the recurring failure classes (https-metadata cert trust, cached :dev image drift → ALWAYS_REBUILD, multi-tenant client_credentials ambiguity, fixed-container-name concurrency), the no-leak Config.Env length-dump diagnostic, and the git blame before theorizing meta-lesson

Plugins, Connectors & Auth (troubleshooting)

• authority-identity-provider-bundles.md — package, sign, mount, restart, and prove LDAP/OIDC/SAML Authority bundles without placing credentials in the bundle or repository
• plugin-cross-alc-missingmethod.md — diagnose a plugin MissingMethodException that is really a cross-ALC type-identity split (bundle ships its own copy of a host-shared contract DLL); the prune fix (package-runtime-plugins.ps1 / prune-concelier-bundle-shared-assemblies.sh), the host-Models.dll-skew and TPA-probe variants, and the swallowed-failure signature
• authority-gateway-auth-troubleshooting.md — recurring deployed-stack auth drifts: http:// Authority metadata → OpenIddict 400 → empty-body 500; CLI password-grant (authority_provider=standard) + empty response.scope; stella-ops-ui audience drift → IDX10214 → console-wide anonymous; standard.yaml not re-merged into the DB
• concelier-connector-operations.md — why a connector “does nothing”: base-vs-recommended profile silent-drop, fixture-mode vs realfeeds, canonical-ingest host bridge, re-map existing edges without re-fetch (pendingMappings + source:map), and the “map succeeded / 0 edges” swallowed-failure case

Conventions

• Each runbook starts with an “Audience” + “Prerequisites” (or equivalent “When to use this” / “When to run”) section.
• Step-by-step procedures use numbered lists with explicit copy-paste commands.
• Verification steps cite expected output (exit code, log line, DB row state).
• Recovery procedures appear under “Failure modes” (or equivalent) at the end.
• Cross-references to ADRs use the relative path ../architecture/decisions/ADR-NNN-*.md.

Related references

• Architecture Decision Records — design decisions that shaped the operations covered here
• Code of Conduct — repo-wide rules including sentinel-grep, testing practices, forward-only migrations
• Implementation Plan (active sprints) and archived sprints — historical context for why operations exist in their current shape