Runbook — Multi-provider secret registry (builtin + openbao + vault)

Audience: Platform / Integrations / ReleaseOrchestrator operators. Applies to: the three services that wire AddRoutingSecretProvider and resolve secret references by URL: StellaOps.Platform.WebService, StellaOps.Integrations.WebService, StellaOps.ReleaseOrchestrator.WebApi. ADR: ADR-032 — URL-based per-resource secret references + multi-provider routing (supersedes ADR-031 §D2). Spec: Secret reference URL scheme. Related: kek-source-vault.md (the master KEK is a separate concern from the secret provider registry).

What this is

ADR-031 bound exactly ONE secret backend per process via Crypto:SecretProvider:Backend. ADR-032 replaces that with a registry of provider instances so a single install can mix builtin + openbao + an external vault simultaneously, and each stored secret reference is a URL that names the provider instance + path. RoutingSecretProvider dispatches every resolve/seal by parsing that URL.

OpenBao positioning (decided 2026-07-04 — ADR-031 §D5 as amended, Option A of the positioning memo): the in-repo OpenBao compose profile is a test-only dev/CI convenience — never shipped or supported in a release. The supported production backends are the built-in floor (the default) and an external, operator-run HashiCorp Vault (the flagship external backend) — or an operator-run OpenBao, since the openbao provider kind reuses the same Vault HTTP client.

The single-backend mode is the degenerate one-entry case: with no Crypto:SecretProviders[] declared, the legacy Crypto:SecretProvider:Backend is honoured as a one-entry back-compat shim — a plain docker compose up (without the override below) behaves exactly as before.

Config shape (ADR-032 §D2)

"Crypto": {
  "SecretProviders": [
    { "id": "builtin", "kind": "builtin" },
    { "id": "bao",     "kind": "openbao", "address": "http://openbao:8200",
      "auth": { "mode": "token", "token": "<op-provided>" } },
    { "id": "vault-prod", "kind": "vault", "address": "https://vault.example:8200",
      "auth": { "mode": "appRole", "roleId": "...", "secretId": "..." } }
  ],
  "SecretProvider": { "DefaultSealTarget": "builtin" }
}

id— the instance segment a reference URL names (openbao://bao/...). Must be unique; the first entry of a kind is that kind’s default instance.
kind— builtin | vault | openbao (unsupported kind ⇒ fail-fast).
address— required for vault/openbao; ignored by builtin.
auth— mode ∈ token | appRole | kubernetes, with the matching fields. No repository default token (fail-closed).
DefaultSealTarget— where NEW secrets seal when the reference URL omits an explicit target. A bare provider id (builtin) or a scheme://id/ URL (openbao://bao/). Omitted ⇒ the first registry entry.

Compose wiring (builtin + test-only openbao — dev/CI)

The registry is expressed in compose via devops/compose/docker-compose.secret-providers.override.yml, which injects the Crypto__SecretProviders__<index>__<field> env vars into all three routing services. The OpenBao instance comes from the test-only Sprint 007 docker-compose.openbao.yml profile (dev/CI convenience — never part of a release; in production, point the openbao/vault entries at an operator-run external broker instead).

# Private .env MUST provide OPENBAO_TOKEN (no repository default; fail-closed):
#   OPENBAO_TOKEN=<operator OpenBao app/root token>
#   OPENBAO_ADDR=http://openbao.stella-ops.local:8200      # default
#   CRYPTO_SECRETPROVIDER_DEFAULTSEALTARGET=builtin        # default

docker compose \
  -f docker-compose.stella-infra.yml \
  -f docker-compose.stella-services.yml \
  -f docker-compose.openbao.yml \
  -f docker-compose.secret-providers.override.yml \
  --profile openbao up -d

Default registry injected: [0] builtin, [1] openbao (id=bao), with DefaultSealTarget=builtin. An external vault-prod entry [2] is included as a commented, opt-in example in the override — uncomment it and supply VAULT_PROD_ADDR + VAULT_PROD_TOKEN (or AppRole) in your .env.

To run single-backend (legacy) mode, simply omit docker-compose.secret-providers.override.yml and leave CRYPTO_SECRETPROVIDER_BACKEND=builtin (or =openbao) — the one-entry shim handles it.

Secret reference URL grammar (quickstart)

Full grammar: secret-reference-url-scheme.md.

<scheme>://[<provider-id>/]<path>[#<version>]
scheme ∈ builtin | vault | openbao | authref (vault alias) | file | base64 | plain

provider-id is optional: omitted ⇒ the default instance of that scheme’s kind; present-but-unknown ⇒ fail-closed (rejected at routing — never a silent fall-through). Examples against the default registry above:

URL	Routes to
`builtin://integration:7b3f…d2`	default builtin, owner-key, current generation
`builtin://integration:7b3f…d2#2`	builtin, generation 2
`openbao://bao/kv/notify/smtp/password`	instance `bao`, OpenBao KV v2
`openbao://kv/notify/smtp/password`	default openbao instance
`vault://vault-prod/secret/app/db/password#5`	instance `vault-prod`, KV version 5
`vault://secret/app/db/token`	default vault instance
`authref://vault/secret/app/db#token`	legacy alias (field via `#`)
`file:///run/secrets/master.key`	inline file (air-gap material)

The URL is non-secret — a pointer, never the value — so it is safe to store, display, log, and audit.

DefaultSealTarget semantics

When an operator seals a NEW secret without naming an explicit scheme://id/ target, RoutingSecretProvider seals it to the provider id resolved from Crypto:SecretProvider:DefaultSealTarget:

bare id (builtin) ⇒ that exact instance;
scheme://id/ URL ⇒ its provider-id segment;
scheme:// with no id ⇒ the first registry entry of that scheme’s kind;
unset ⇒ the first registry entry.

A bare-id DefaultSealTarget that names no configured provider is rejected at config bind (fail-closed) — see SecretProvidersOptions.Validate.

Fail-closed posture (ADR-031 §D3, ADR-032 §D3)

No repository default tokens. OPENBAO_TOKEN (and any external VAULT_PROD_TOKEN) use the compose :? required-variable form — a fresh docker compose config/up errors loudly when unset rather than running with a known credential.
An unresolvable reference (unknown scheme, unknown provider-id, absent provider for a required seal) is rejected loudly by the router — never a silent downgrade. A recognised-but-absent value still returns null (the ADR-031 contract is preserved).

Master key at setup (the seal/open root)

The provider registry (above) decides where a secret lives. The master key (KEK) decides what seals the builtin floor and what wraps externally sourced material — it is a separate concern (ADR-031 §D3, §D6; cross-ref kek-rotation.md, kek-source-vault.md, kek-source-hsm.md).

The setup wizard exposes a Master Key step (SetupStepId.MasterKey) served by GET /api/v1/setup/definitions/steps. The step gates on a probe — a round-trip seal/open against the resolved KEK — so boot fails closed (the legacy “key absent until first seal/open” lazy failure is removed):

# Probe-only seal/open round-trip against the resolved master key (no key change):
POST /api/v1/admin/setup/crypto/master-key/probe     # crypto:kek:rotate + tenant
#   → { "data": { "ok": true, "message": "Master key seal/open round-trip succeeded." } }

# Provision a NEW master key at first setup (symmetric default, or asymmetric RSA/EC):
POST /api/v1/admin/setup/crypto/master-key/generate  # crypto:kek:rotate
POST /api/v1/admin/setup/crypto/master-key/import    # operator-provided material

WARNING — never rotate the live master key casually. Generating, importing, or rotating the active installation KEK re-roots seal/open and will orphan every existing sealed secret (connector creds, deployment bundles) unless a full re-seal walk is run (see kek-rotation.md). On a running, data-bearing install, use the probe to verify the key is healthy; only generate/import on a fresh install or under a planned rotation.

Crypto-control surface (Console)

The Console admin crypto-control page renders the live secret-provider state for operators without shell access:

Secret Provider panel — backend (default seal target), reachability, the active KEK id + fingerprint (non-secret), and the default AEAD algorithm.
Validate provider — runs the same probe-only seal/open round-trip (GET …/setup/crypto/secret-provider/validate → outcome:ok, masterKeyPresent:true, kekFingerprint:…). GREEN = key present + backend reachable.
SecretReferenceInputComponent— wherever a secret is configured (registry credentials, deployment selectors), the operator gets a Select… picker that lists the configured providers with reachable badges, then a path autocomplete that searches the provider for real path names (names only — never values). The field stores the resulting URL reference, broken down as scheme · provider · path · field, so the stored value is always a pointer.

CLI surface

stella crypto secret-provider (read-only inspection; crypto:read) and stella crypto master-key (provisioning; crypto:kek:rotate). Full reference: CLI crypto command guide.

stella crypto secret-provider list                         # id / kind / reachable
stella crypto secret-provider status                       # backend + KEK fingerprint (non-secret)
stella crypto secret-provider validate                     # master key present + reachable
stella crypto secret-provider paths <id> [--prefix P]      # browse path NAMES only
stella crypto master-key probe                             # seal/open round-trip (setup gate)

secret-provider verbs never print secret bytes (paths returns names only; JSON output has no value field). paths <unknown-id> fails closed with HTTP 404 listing the configured ids (exit 14). master-key generate/import write/forward key material out-of-band (asymmetric private key → owner-only file; import material read from a file, never a CLI arg) and are gated on crypto:kek:rotate.

Verify

# The /secret-providers admin endpoint enumerates the configured registry +
# per-provider describe state (ADR-032 §D5):
GET /api/v1/admin/crypto/secret-providers          # crypto:read + tenant

# docker compose config sanity (registry injected into all three services):
docker compose -f docker-compose.stella-infra.yml \
  -f docker-compose.stella-services.yml \
  -f docker-compose.openbao.yml \
  -f docker-compose.secret-providers.override.yml \
  --profile openbao config | grep Crypto__SecretProviders__

Proven live (2026-06-22)

This registry, the master-key probe gate, the crypto-control surface, and the CLI verbs were proven end-to-end against a live, data-bearing stack (builtin + 2× HashiCorp Vault + OpenBao, all reachable) without touching the live master key. Full evidence (API + UI screenshots + CLI transcripts): docs/qa/feature-checks/runs/2026-06-22-secret-provider-live/.

In that proof, OpenBao was a TEST-ONLY backend (ephemeral container via the opt-in docker-compose.openbao.yml profile), not a shipped default. That is now the decided, permanent positioning (2026-07-04 — ADR-031 §D5 as amended): the product default stays builtin; the supported external backend is an operator-run Vault (or operator-run OpenBao); the in-repo OpenBao profile is dev/CI test tooling only.

ADR-032 — URL secret references: ../architecture/decisions/ADR-032-url-secret-references.md
ADR-031 — unified secret provider: ../architecture/decisions/ADR-031-unified-secret-provider.md
URL scheme spec: ../implplan/specs/secret-reference-url-scheme.md
Override: devops/compose/docker-compose.secret-providers.override.yml
OpenBao profile: devops/compose/docker-compose.openbao.yml
Routing source: src/__Libraries/StellaOps.Cryptography.CredentialStore.Vault/RoutingSecretProviderServiceCollectionExtensions.cs
Options/binding: src/__Libraries/StellaOps.Cryptography.CredentialStore/SecretProvidersOptions.cs