ADR-012: Per-tenant system-account client_credentials clients auto-seeded during tenant onboarding (Pattern C)

Status: Accepted Date: 2026-05-29 Sprint: SPRINT_20260529_053_Authority_per_tenant_client_autoseed_onboarding.md (S053-001) Related ADRs: ADR-005 (single-operator multi-environment tenancy), ADR-006 (execution plugin framework), ADR-007 (KEK lifecycle and rotation — same Vault substrate), ADR-010 (shared tenant resolver migration) Related sprints: Sprint 028 RP-028-008 (HttpVaultKvReader — wire-level Vault HTTP client). Sprint 031 (archived; tenant-onboarding ITenantClientGrantService + bootstrap merge of properties.tenants). Sprint 048 S048-002 (commit c1bef51ab5; YAML + idempotent SQL backstop seed-template). Sprint 051 (Pattern B; lab/dev multi-tenant client — coexists with Pattern C during prod rollout). Related code: src/Authority/StellaOps.Authority/StellaOps.Authority/Console/Admin/ConsoleAdminEndpointExtensions.cs:315-431 (CreateTenant handler — the Pattern C insertion point), src/Authority/StellaOps.Authority/StellaOps.Authority/Tenants/TenantClientGrantService.cs (Sprint 031 auto-grant primitive — Pattern C reuses the call-site, not the merge semantics), src/Authority/StellaOps.Authority/StellaOps.Authority/OpenIddict/Handlers/ClientCredentialsHandlers.cs:1841-1862 (the singular-tenant binding handler — Pattern C clients USE this binding deliberately), src/Authority/StellaOps.Authority/StellaOps.Authority.Plugin.Standard/Storage/StandardClientProvisioningStore.cs:100-211 (ApplyRegistration — Pattern C clients are NOT in standard.yaml; the bootstrap reseed never touches them), src/__Libraries/StellaOps.Cryptography.CredentialStore.Vault/HttpVaultKvReader.cs (existing read surface; Pattern C extends with a write surface), src/Scanner/StellaOps.Scanner.Worker/Processing/ConcelierLearnTokenHandler.cs (per-tenant token cache shape Pattern C generalises).

In one line: at prod scale, give every Stella Ops tenant its own auto-seeded Authority client_credentials identity for system-account services (scanner-worker first), with the secret in HashiCorp Vault — so a leaked secret can never mint tokens for any other tenant.

Audience: Authority and platform engineers implementing tenant onboarding, and operators reasoning about per-tenant blast-radius containment. Builds on ADR-005 (tenancy) and ADR-007 (the Vault substrate).

Context

Stella Ops is moving from a few hand-onboarded tenants (lab + lab-like dev) to prod-scale tenant onboarding where each new customer tenant is created via an admin endpoint (Sprint 031’s POST /console/admin/tenants). The scanner-worker (and other “system-account” services that fan-out per-tenant work — Concelier learn, integrations resolver, attestor, etc.) needs an Authority client_credentials identity per tenant in order to mint tenant-scoped tokens that resource servers gate on (stellaops:tenant claim).

Sprint 046 + 019 surfaced the friction: the single seeded stellaops-cli-automation client inherited tenantId: "default" from etc/authority/plugins/standard.yaml:4 and the ClientCredentialsHandlers.cs:1855 guard rejected any other selectedTenant. Three patterns were considered post-H4 architectural analysis (2026-05-29):

Operator decision 2026-05-29 (post-H4): “chain 2 → go B+C”. Pattern B (Sprint 051) ships immediately as the tactical Sprint 046/019 unblock for lab/e2e-lab. Pattern C (this ADR) is the prod-scale long-term path that retires the multi-tenant blast-radius concern via per-tenant containment.

This ADR is the architectural design for Pattern C. The execution roadmap (S053-002) and per-phase execution sprints follow when prod-scale tenant onboarding becomes a real driver (currently lab/e2e-only; Pattern B suffices).

Existing infrastructure Pattern C builds on

SurfaceFileRole for Pattern C
Tenant-onboarding endpointConsoleAdminEndpointExtensions.cs:315-431Insertion point: extend the CreateTenant handler to atomically issue + Vault-store per-tenant scanner-worker client alongside tenantRepository.CreateAsync + PropagateSharedTenantAsync + tenantClientGrantService.GrantAsync.
Tenant-client grant primitiveTenants/TenantClientGrantService.cs (Sprint 031)Reuse the call-site pattern (post-propagation, non-fatal-on-failure, audit-property capture). Pattern C does NOT extend the Sprint 031 primitive — it adds a sibling primitive IPerTenantSystemAccountProvisioner because the semantics are different (issue a new client, not merge into an existing one).
Singular tenant bindingClientCredentialsHandlers.cs:1841-1862Pattern C clients SET descriptor.Tenant = <tenant> deliberately. The guard at line 1855 becomes the per-tenant containment fence: any token request with a selectedTenant other than the descriptor’s tenant is rejected with invalid_client. Pattern B suppresses this binding; Pattern C requires it.
Vault KV read surfaceHttpVaultKvReader.cs (Sprint 028 RP-028-008)Reused verbatim by scanner-worker secret resolution. The reader handles KV v2, three auth modes (token / approle / kubernetes), 80% lease re-auth, and VaultPermanent vs VaultUnavailable exception split.
Vault KV write surface(does not exist yet)New: IVaultKvWriter interface + HttpVaultKvWriter implementation in the same library (StellaOps.Cryptography.CredentialStore.Vault). Mirrors the reader’s wire-level approach (no VaultSharp dependency) — single endpoint POST /v1/<mount>/data/<path>. Required by Authority for secret-write on onboarding + rotation; required by tenant-decommission for secret-delete.
Token-cache-per-tenant shapeConcelierLearnTokenHandler.cs (Sprint 022 002)Generalise ConcurrentDictionary<string,CachedToken> keyed by tenant. Pattern C extends to (tenant, audience) key + adds Vault-resolved secret lookup before RequestClientCredentialsTokenAsync so each per-tenant client’s secret enters the token request from Vault, not from process-resident config.
IAuthorityClientStoreStellaOps.Authority.Persistence/Stores/IAuthorityClientStore.csReused for the actual client document write. Pattern C adds a deterministic clientId = stellaops-scanner-worker-<tenant> + per-tenant secret hash (via AuthoritySecretHasher.ComputeHash); the secret plaintext goes to Vault, never to authority.clients.client_secret.

Why Pattern B and Pattern C must coexist (not replace each other)

Pattern B retires per-environment as Pattern C lands in that environment. Lab + dev keep Pattern B during the prod-scale rollout because (a) Pattern B has zero per-tenant operational cost and lab tenants come and go too frequently to amortise per-tenant Vault writes, and (b) the lab’s blast-radius tolerance is fundamentally higher than prod’s. The migration from Pattern B to Pattern C in a given environment is a single operator step (Phase 5 below): remove the multi-tenant stellaops-cli-automation client’s tenants plural, run onboarding for each existing tenant. No data migration; no token-cache flush in the consuming services (they observe both client_ids during the cutover).

Decision

Adopt Pattern C. Extend the tenant-onboarding workflow (Sprint 031’s CreateTenant) to atomically issue a per-tenant stellaops-scanner-worker-<tenant> client_credentials client with a Vault-stored per-tenant secret. The scanner-worker (and future system-account services) resolves the per-tenant secret from Vault at token-acquire time, with a per-(tenant, audience) token cache and fail-closed semantics when Vault is unreachable for the tenant.

Per-tenant client shape (target)

clientId        = "stellaops-scanner-worker-<tenant-slug>"
displayName     = "Stella Ops Scanner Worker — <tenant-slug>"
confidential    = true
clientSecret    = <generated at onboarding; 32-byte CSRNG → base64url, ≥256 bits entropy>
secretHash      = AuthoritySecretHasher.ComputeHash(clientSecret)   // persisted to authority.clients
allowedGrantTypes  = ["client_credentials"]
allowedAudiences   = ["api://integrations", "stellaops"]            // matches Sprint 048 S025
allowedScopes      = [<scanner-worker subset; see "Scope set" below>]
tenant          = "<tenant-slug>"                                    // singular, deliberate (Pattern C contract)
properties.tenants = "<tenant-slug>"                                  // singular, mirrors tenant
properties.system_account = "scanner-worker"                          // tag for inventory + rotation policy lookup
properties.provisioned_by = "tenant-onboarding-v1"                    // provenance for migrate-from-Pattern-B detection
properties.vault_secret_path = "stellaops/tenants/<tenant-slug>/system-accounts/scanner-worker"

tenant-slug constraint. Pattern C reuses the tenant-onboarding regex (ConsoleAdminEndpointExtensions.cs:332 — lowercase letters, digits, hyphens only). Slugs are bounded ≤ 63 chars (matches shared.tenants.slug); the derived clientId is therefore bounded ≤ len("stellaops-scanner-worker-") + 63 = 88 chars, well under any authority.clients.client_id column limit.

Scope set. Minimum scanner-worker subset (intentionally narrower than stellaops-cli-automation’s full operator scope set):

aoc:verify advisory:ingest advisory:read concelier.jobs.trigger concelier.merge
graph:read integration:read integration:write integration:operate
registry.admin scanner:read scanner:scan scanner:write scanner:export
sbom:read vex:read vexhub:read findings:read

Operator extensibility: the scope set is captured in a single config key (Authority:PerTenantSystemAccounts:ScannerWorker:AllowedScopes — see “Config shape” below) so future scanner-worker scope additions (e.g., a new learn:operate scope) propagate to every tenant on the next Authority restart without per-tenant migration.

Vault path scheme (canonical, on-prem-first)

mount = secret                                                                        # KV v2 default mount; operator-overridable
path  = stellaops/tenants/<tenant-slug>/system-accounts/<account-name>                # account-name = "scanner-worker" today; "attestor", "advisory-ai", … future
field = client_secret                                                                 # opaque base64url string

Why this scheme:

Operator-overridable. Like the KEK source (ADR-007 → Crypto:Kek:Vault:Path), Pattern C’s path template is configurable (Authority:PerTenantSystemAccounts:Vault:PathTemplate = "stellaops/tenants/{tenant}/system-accounts/{account}") for operators with existing path conventions. The template uses named placeholders, not positional, so the order can’t drift.

Vault write surface (new — IVaultKvWriter / HttpVaultKvWriter)

Add to StellaOps.Cryptography.CredentialStore.Vault. Wire-level KV v2 write (POST /v1/<mount>/data/<path>), same auth-mode + lease + exception split as HttpVaultKvReader. Returns the Vault version number on success (for audit + idempotency: if a write fails mid-rotation, the operator can verify whether the new version landed).

public interface IVaultKvWriter
{
    Task<int> WriteFieldAsync(string mount, string path, string field, string value, CancellationToken ct);
    Task DeleteAsync(string mount, string path, CancellationToken ct);   // KV v2 metadata destroy (full removal, not soft-delete)
}

Two consumers in the Pattern C design:

  1. Authority (during onboarding/rotation): writes the freshly generated secret atomically inside the onboarding transaction.
  2. Authority (during tenant-decommission): hard-deletes the Vault entry via KV v2 metadata destroy so the secret cannot be recovered from Vault version history.

Scanner-worker has read-only Vault access (the existing IVaultKvReader). The write surface lives in Authority’s network footprint exclusively.

Atomic onboarding (rollback semantics)

Extend CreateTenant (ConsoleAdminEndpointExtensions.cs:315-431) with a new step between PropagateSharedTenantAsync (line 380) and the existing tenantClientGrantService.GrantAsync (line 395):

[existing] tenantRepository.CreateAsync                      // step 1: authority.tenants row
[existing] PropagateSharedTenantAsync                        // step 2: shared.tenants upsert
[NEW]      perTenantSystemAccountProvisioner.ProvisionAsync  // step 3: per-tenant client + Vault secret (atomic)
[existing] tenantClientGrantService.GrantAsync               // step 4: extend stellaops-cli + others (Sprint 031)
[existing] WriteAdminAuditAsync                              // step 5: audit

IPerTenantSystemAccountProvisioner.ProvisionAsync(tenantSlug, ct) is the atomic unit. It performs:

  1. Generate secret in-process (32 bytes CSRNG → base64url).
  2. Vault write (IVaultKvWriter.WriteFieldAsync) — secret lands first; this is the “cannot fully undo” step from Authority’s perspective. If it fails: throw; nothing downstream has happened; tenant create proceeds with a non-fatal warning audit (see “Failure modes” below).
  3. Authority client write (IAuthorityClientStore.UpsertAsync) with secretHash computed via AuthoritySecretHasher.ComputeHash. If it fails: Vault entry is orphan — best-effort IVaultKvWriter.DeleteAsync rollback in the catch handler; if THAT fails too, log a tenant.system_account.orphan_vault_secret event with the path so the operator can clean up. The orphan is harmless (no client references it) but operator-visible.
  4. Audit-property capture — record system_account.provisioned, system_account.vault_path, system_account.client_id, system_account.outcome on the tenant-create audit event. NO secret material in any audit field.

Why Vault write FIRST, Authority client write SECOND. The reverse ordering would leave a client with a hash that no Vault entry can satisfy → scanner-worker token-acquire fails fail-closed → noisy but recoverable. The chosen ordering leaves at worst an orphan Vault entry (no client references it; no client can use it) → silent and equally recoverable. The chosen ordering is the cleaner default because the Authority secret_hash is the harder-to-rotate side: rewriting Vault is a single API call; rewriting authority.clients.secret_hash requires a new client-update endpoint OR a forward-only migration. Keeping the harder side last shrinks the abort window.

Non-fatal on provisioning failure. Mirroring PropagateSharedTenantAsync’s posture (Sprint 031’s “tenant exists; propagation is the post-fix”): if ProvisionAsync throws, the tenant-create still returns 201 Created with a Warning response header Warning: 199 - "scanner-worker system account provisioning failed; run \stella system provision-account` to remediate". The tenant is operational for everything that doesn't need scanner-worker; the operator gets a clear remediation path. This is the same posture Sprint 031 chose forITenantClientGrantService(ConsoleAdminEndpointExtensions.cs:395-403` comment).

Scanner-worker per-tenant secret resolution

Generalise ConcelierLearnTokenHandler.cs into a reusable PerTenantSystemAccountTokenHandler in StellaOps.Auth.Client. Key shape change vs. Sprint 022’s handler:

// Today:  ConcurrentDictionary<string, CachedToken>                          keyed by tenant
// New:    ConcurrentDictionary<(string Tenant, string Audience), CachedToken>

SendAsync resolves the per-tenant clientId (stellaops-scanner-worker-<tenant>) + Vault-resolved secret (IVaultKvReader.ReadFieldAsync("secret", $"stellaops/tenants/{tenant}/system-accounts/scanner-worker", "client_secret", ct)), then calls RequestClientCredentialsTokenAsync with the resolved secret. Per-(tenant, audience) cache. Tokens are cached for 0.8 × expires_in (matching the existing 30-second RefreshSkew).

Fail-closed on Vault unreachable for a tenant. If IVaultKvReader.ReadFieldAsync throws VaultUnavailableException, the handler does NOT fall back to any other secret source — it returns 503 to the caller (or fails the per-tenant scan job, depending on the dispatch path) with failure_mode = "vault_unreachable". If it throws VaultPermanentException (404, 403, malformed payload), it returns 503 with failure_mode = "vault_permanent". Both failure modes are operator-visible per-tenant; one tenant’s Vault outage does NOT degrade the others (the cache + per-tenant resolution scope keeps blast radius bounded).

Secret-rotation path

Two cadences:

  1. Scheduled rotation (90-day default; operator-configurable per-tenant via Authority:PerTenantSystemAccounts:ScannerWorker:RotationCadenceDays). A new admin endpoint POST /console/admin/tenants/{tenantId}/system-accounts/{account}/rotate (or the equivalent stella system rotate-account <tenant> <account> CLI command):

    • Generate new secret in-process.
    • Vault write (new version; KV v2 keeps the prior version for grace-window read-back if needed).
    • Authority client update (replace secretHash).
    • Audit event authority.system_account.rotated with tenant, account, kek_version_n_minus_1 for forensic trail.
    • Scanner-worker token cache invalidation is not required — the cache holds tokens, not secrets; the next cache miss (≤ token TTL) re-reads Vault. Operators who want zero-grace-window rotation can call the future IPerTenantSystemAccountTokenHandler.InvalidateAsync(tenant) API (deferred to a follow-up sprint; not required for the 90-day cadence).

  2. Emergency rotation (suspected leak). Same endpoint with ?reason=leak flag → audit captures the operator-supplied reason; otherwise identical to scheduled rotation. The fact that the token cache lives in scanner-worker process memory means the worst-case “stolen secret” exposure window is (Vault write time) + (worker token TTL ≤ 1h). Operators who need sub-TTL invalidation use the future cache-invalidation API.

No double-write window. Authority does not support two valid secrets per client simultaneously (the schema has a single secret_hash column). Rotation is therefore a replace, not an add-then-remove. Acceptable because: (a) scanner-worker re-reads Vault on cache miss (so the new secret propagates ≤ token TTL after rotation), and (b) the only consumer is the per-tenant scanner-worker process, with no external integrations holding the secret.

Tenant-decommission path

There is NO DELETE /tenants/{id} endpoint today (verified at ConsoleAdminEndpointExtensions.cs:104 — only the operator-compliance-approver subresource has MapDelete). When the future tenant-delete endpoint lands, it MUST invoke (in this order):

  1. IPerTenantSystemAccountProvisioner.DecommissionAsync(tenantSlug, ct):
    • IAuthorityClientStore.DeleteAsync(clientId) — removes the client row + secret hash from Authority.
    • IVaultKvWriter.DeleteAsync(mount, path, ct) — KV v2 metadata destroy (full removal, not soft-delete; the leaked-secret hypothesis means version history must not be recoverable).
    • PerTenantSystemAccountTokenHandler.InvalidateAsync(tenant) on the broadcast — any in-flight scanner-worker token cache entry for the decommissioned tenant is dropped.
  2. tenantClientGrantService.RevokeAsync(tenantSlug) — Sprint 031 primitive; removes the tenant from stellaops-cli + other auto-grant clients.
  3. sharedTenantsPropagator.RemoveAsync (TBD when the propagator gains a remove surface).
  4. tenantRepository.DeleteAsync.
  5. Audit authority.admin.tenants.delete with system_account.decommissioned, system_account.vault_deleted, system_account.client_deleted properties.

Until that endpoint exists, a tenant suspended via the existing suspend endpoint MUST preserve its system-account client + Vault entry (resume must restore exactly the same posture; matching Sprint 031’s grant-survives-restart property).

Config shape

New configuration section:

Authority:
  PerTenantSystemAccounts:
    Enabled: true                                            # master switch; default true; false = behaviour-preserving (no provisioning, no error)
    Vault:
      Address: "https://vault.internal:8200"                 # reuses ADR-007's Vault server by default; per-section override allowed
      Mount: "secret"
      PathTemplate: "stellaops/tenants/{tenant}/system-accounts/{account}"
      Field: "client_secret"
      Auth:                                                  # mirrors VaultAuthOptions; same modes (token / approle / kubernetes)
        Mode: "approle"
        RoleId: "<approle role id for Authority writes>"
        SecretId: "<approle secret id for Authority writes>"
      WriteRetryWindow: "00:01:00"                           # transient Vault unavail → retry up to 1 minute before failing onboarding step
    Accounts:
      ScannerWorker:
        ClientIdPrefix: "stellaops-scanner-worker-"
        AllowedAudiences: ["api://integrations", "stellaops"]
        AllowedScopes: [<see "Scope set" above>]
        RotationCadenceDays: 90
        SecretEntropyBits: 256                               # CSRNG → 32 bytes → base64url
      # future: Attestor, AdvisoryAI, etc. — same shape, different ClientIdPrefix + scope set

The Accounts.<Account> shape is open: adding a per-tenant Attestor client to the design is a config-only change (Phase 6 — out of scope for this ADR). The scanner-worker shape is the only one this ADR mandates.

Failure modes (summary table)

ModeWhat happensTenant-create outcomeOperator surface
Vault unreachable at onboardingIVaultKvWriter.WriteFieldAsync throws VaultUnavailableException after WriteRetryWindow201 Created + Warning: 199 headerstella system provision-account <tenant> remediation; authority.system_account.provision_failed audit
Vault permanent error (403/404/payload) at onboardingThrows VaultPermanentException201 Created + Warning: 199 headerSame remediation; audit captures failure_mode = "vault_permanent"
Authority client write fails after Vault wroteBest-effort IVaultKvWriter.DeleteAsync rollback201 Created + Warning: 199 headerSame remediation; if rollback also fails, authority.system_account.orphan_vault_secret audit captures the path
Vault unreachable at token-acquireScanner-worker returns 503 for that tenant(not a tenant-create event)Per-tenant scan jobs fail-closed until Vault recovers; other tenants unaffected
Vault permanent error at token-acquireScanner-worker returns 503 for that tenant(not a tenant-create event)failure_mode = "vault_permanent" — usually a Vault policy drift; operator restores read capability on stellaops/tenants/<tenant>/system-accounts/scanner-worker
Authority client deleted but Vault entry survivesToken-acquire returns invalid_client from Authority(not a tenant-create event)Same operator surface as decommission cleanup; orphan Vault entry harmless

Sentinel-leak posture

The provisioning code path generates the secret in-process, writes it once to Vault and once (hashed) to Authority, and discards the plaintext before any audit/log/HTTP-response emission. Sentinel-leak test (mandatory per the Sprint 031 precedent): inject a known-distinguishable secret pattern (SENTINEL_S053_***) into the CSRNG seam, exercise ProvisionAsync, assert the pattern does not appear in any captured log/audit/HTTP-response payload. The same posture applies to rotation.

Consequences

Positive

Tradeoffs

Alternatives Considered

Pattern A — per-tenant manually seeded clients in standard.yaml

Rejected. Pros: no Vault dependency; secret hashes live in Authority alongside every other client. Cons: every new tenant becomes a YAML edit + Authority migration + service restart. Scales sub-linearly with tenant count: each new tenant is a manual operator step with no automation surface. Operator-confirmed rejection 2026-05-29 as a “scaling smell” in the H4 architectural analysis.

Pattern B (alone) — multi-tenant single client + properties.tenants allowlist

Rejected for prod. Pros: cheapest possible; existing infrastructure; no new code beyond a YAML edit. Cons: single secret = blast-radius across every allowed tenant; leak = attacker-as-every-allowed-tenant. Acceptable for lab/dev (Sprint 051 ships it). Inappropriate for prod where per-tenant blast-radius containment is the table-stakes security property.

Per-tenant clients with secret in Postgres (no Vault)

Rejected. Pros: no new infrastructure dependency. Cons: violates feedback_on_prem_first_no_cloud_defaults’s Vault-flagship principle (the spirit of the rule is “secrets at rest live in Vault, not in service databases”); puts plaintext-equivalent material (the hash + the rotation cadence + the operator-visible audit) in the same database as the tenant-onboarding workflow that emits it (no defence-in-depth); makes secret rotation a multi-table transaction instead of a single Vault API call. The Vault path is the architecturally correct location for per-tenant secret material in a Stella Ops on-prem deployment.

Per-tenant clients with secret in a Stella-Ops-managed credential store (not Vault)

Rejected. Same reasoning as the Postgres alternative + adds a new credential-store substrate to maintain. Vault already exists; reuse the existing substrate.

Auto-seed at Authority startup (not at tenant onboarding)

Rejected. Pros: removes the Vault dependency from the onboarding critical path. Cons: a tenant created mid-day would have to wait for the next Authority restart to get a system account → defeats the “zero manual operator step” goal. Onboarding-time provisioning is the right insertion point because the tenant is by definition new (no pre-existing system account to reconcile against) and the operator is already mid-flow.

Migration steps (handed off to S053-002)

Sprint 053 S053-002 (below in the sprint dossier) sequences this design into per-phase execution sprints with acceptance gates. Each phase is a separate executable sprint, dispatched when prod-scale tenant onboarding becomes a real driver.

References