Runbook — Credential storage algorithm switch (per-tenant)

Switch a tenant’s at-rest credential-encryption algorithm (for example from the default AES-256-GCM world posture to a regional posture such as SM/ShangMi or GOST) and re-seal existing credential ciphertext under the new algorithm.

Audience: Platform / Integrations / ReleaseOrchestrator operators in regulated regions (eIDAS, FIPS, GOST R, SM/ShangMi). Sprint: SPRINT_20260529_028 (RP-028-014 — sister of RP-028-009…013). Applies to: the three credential-ciphertext stores (platform.connector_credentials, integrations.integrations, release_orchestrator.deployment_bundles). Related ADRs: ADR-008 (regional algorithm routing), ADR-007 (KEK lifecycle). Related runbooks: kek-rotation.md, kek-source-vault.md, kek-source-hsm.md.

When to switch

A per-tenant credential storage algorithm switch is the right move when:

The algorithm choice is per-tenant, not per-installation: different tenants on the same install can carry different algorithms simultaneously, because the dispatch layer (ICredentialAlgorithmSelector reading the tenant’s compliance profile + the per-row algorithm column + ICredentialAeadPrimitive) selects the algorithm per row by that row’s tenant. (The per-row algorithm column name differs by store: secret_alg in platform.connector_credentials, credential_alg in integrations.integrations, and alg in release_orchestrator.deployment_bundles.)

Important — the re-seal walker is NOT tenant-scoped. stella crypto profile set <p> --reseal --tenant <id> does two distinct things: (a) the --tenant flag scopes the profile change to that tenant; (b) --reseal then triggers a KEK rotation across the entire kek_id store (the walker batches WHERE kek_version = oldVersion with no tenant predicate — see CredentialReSealWalker / ReSealRequest). Every row on that store is re-sealed, and each row’s new algorithm is whatever the selector returns for that row’s tenant. So a row owned by a tenant still on world is re-sealed back under AES-256-GCM in the same pass, while the just-switched tenant’s rows move to the regional algorithm. The net effect is the desired per-tenant outcome, but the operator should understand the walker touches all tenants’ rows for the kek_id, not only the named tenant’s.

Prerequisites

Required scopes. The operator’s token (or the stella CLI’s resolved credentials) must carry the right Platform crypto scopes — each is also satisfied by ops.admin:

  • crypto:profile:admin — set the tenant compliance profile (PUT /api/v1/admin/crypto-providers/compliance-profile, policy platform.crypto.profile.admin). NB: the read-only validate verb (GET /api/v1/admin/crypto/profile/validate) lives under a different path prefix and a different scope (crypto:read, see below).
  • crypto:kek:rotate — begin/abort the re-seal walker (POST .../kek/{id}/rotate, policy platform.crypto.kek.rotate).
  • crypto:read — stella crypto profile validate / show (policy platform.crypto.read).
  • crypto:kek:read — stella crypto kek status / history (also satisfied by crypto:kek:rotate; policy platform.crypto.kek.read).

Scope constants are canonical in src/Authority/.../StellaOps.Auth.Abstractions/StellaOpsScopes.cs (crypto:profile:admin, crypto:kek:rotate, crypto:read, crypto:kek:read); the Platform policy registrations live in src/Platform/StellaOps.Platform.WebService/Program.cs.

  1. The correct regional plugin must be loaded and registered. When PrimitiveRouter=plugin is set, Platform eagerly initializes the bundled SmPlugin / FipsPlugin / GostPlugin on first resolve (see RegionalCryptoPluginServiceCollectionExtensions); init failures bubble as a startup error rather than silently falling back to AES. Inspect the bootstrap log — the regional plugins log under the RegionalCryptoPlugin.Sm / RegionalCryptoPlugin.Fips / RegionalCryptoPlugin.Gost categories, and the generic loader logs Loading crypto plugin manifest from: / Failed to load crypto plugins:

    docker logs platform | grep -iE "RegionalCryptoPlugin|crypto plugin"

    A clean boot with the plugin router enabled means the adapters (Sm4GcmAdapter / AesGcmFipsAdapter / GostGcmAdapter) are wired into the PluginRoutedCredentialAeadPrimitive. Note: there is no separate StellaOps.Cryptography.Plugin.Sm external plugin DLL to discover here — the regional plugins are bundled and DI-registered in-process.

  2. Plugin routing must be enabled (SPRINT_20260528_035 / S035-003). Default DI on Platform binds the AES-only path so installs without regional crypto requirements keep Sprint 027 byte-identity untouched. Two opt-in flags activate the regional dispatch — wire BOTH for the full per-tenant-profile path:

    {
  "Crypto": {
    "CredentialStore": {
      "PrimitiveRouter": "plugin",
      "Selector": "tenant-compliance"
    }
  }
}

    What each flag changes (Platform Program.cs):

    FlagEffect when unsetEffect when set
    Crypto:CredentialStore:PrimitiveRouter = pluginICredentialAeadPrimitive → IAeadAlgorithmCredentialPrimitive (AES-only via IAeadAlgorithm)ICredentialAeadPrimitive → PluginRoutedCredentialAeadPrimitive with SmPlugin / FipsPlugin / GostPlugin adapters wired
    Crypto:CredentialStore:Selector = tenant-complianceICredentialAlgorithmSelector → DefaultCredentialAlgorithmSelector (returns AES-256-GCM for every tenant)ICredentialAlgorithmSelector → TenantComplianceProfileCredentialAlgorithmSelector (consults platform.tenant_compliance_profile → CredentialAlgorithmProfileMap)

    Equivalent env-var form for docker-compose .env:

    Crypto__CredentialStore__PrimitiveRouter=plugin
Crypto__CredentialStore__Selector=tenant-compliance

    Accepted values (case-insensitive):

    • PrimitiveRouter: default / aead / iaeadalgorithm (AES-only), plugin / plugins / router (regional). Any other value = fail-fast at boot (no silent downgrade — a typo on a regulated install must not silently revert to AES).
    • Selector: default or empty (AES-only), tenant-compliance / tenantcompliance (consult tenant compliance profile). Any other value = fail-fast at boot.

    Restart Platform after changing either flag — the primitive + selector are bound at DI composition.

    Integrations + ReleaseOrchestrator scope. S035-003 wires these flags on Platform only. Integrations + RO build their own CredentialAead instances directly with IAeadAlgorithm (distinct aadPrefix + kdfLabel, mutually undecryptable with Platform’s ciphertexts) and do NOT currently bind ICredentialAeadPrimitive or ICredentialAlgorithmSelector in DI — so the conditional opt-in has nothing to swap there. Their local service-secret stores therefore continue on the Sprint 027 AES-256-GCM path; mirroring the regional dispatch into those services is a separate ladder (each service’s credential surface is much smaller than Platform’s connector store, and the regulated-customer requirement starts at the connector store per ADR-008). The live forcing-function smoke (S035-004 dossier) targets Platform’s platform.connector_credentials table accordingly.

  3. stella crypto profile validate must return okunder the target profile. The verb validates the profile of the active request tenant (resolved server-side from the auth/tenant context by GET /api/v1/admin/crypto/profile/validate) — it does not accept a --tenant argument. To validate a different tenant, invoke the CLI with that tenant’s credentials/context. Use --format json for the machine shape:

    stella crypto profile validate --format json

    Expected output (the server wire shape, lowerCamelCase):

    {
  "outcome": "ok",
  "activeProfile": "sm",
  "activeAlgorithm": "SM4-GCM",
  "primitiveKind": "plugin_routed",
  "pluginReachable": true,
  "fipsModeDisagreement": false,
  "notes": []
}

    (primitiveKind is default when the AES-only IAeadAlgorithmCredentialPrimitive is bound and plugin_routed when the plugin router is active.) The endpoint’s outcome is one of ok, profile_inactive, no_plugin_matches, fips_disagreement, roundtrip_failed. If the outcome is no_plugin_matches, the configured primitive does not support the profile’s algorithm (regional plugin not registered or no adapter wired). If fips_disagreement, the active profile is fips but the host is not FIPS-validated. (As of RP-028-013 the Platform FIPS-mode probe IsFipsProfileMisaligned() returns false conservatively — a live fips_disagreement requires the FipsPlugin’s cross-platform probe, which is kept out of the Platform assembly; the dossier test cases force the path directly.) Fix the underlying issue before switching — the walker will fail mid-way otherwise.

  4. KEK source healthy. The walker re-seals every row, which means it reads the OLD ciphertext and writes the NEW ciphertext, both under the active KEK. If the KEK source is unhealthy, the walker will fail; run stella crypto kek status first.

Switch workflow

1. Persist the tenant profile + trigger walker — single CLI invocation

stella crypto profile set sm --reseal --tenant <tenant-id>

What this does:

  1. Persists the new compliance profile (sm here) via the tenant compliance profile store (ITenantCryptoComplianceClient.SetProfileAsync).
  2. Maps the profile to the canonical algorithm (MapProfileToAlgorithm):
    • sm → SM4-GCM
    • gost → GOST-GCM
    • fips / eidas / world / kcmvp / unset → AES-256-GCM
  3. Invokes IPlatformCryptoKekClient.BeginRotationAsync(kekId, targetAlgorithm="SM4-GCM") — POST .../kek/{kekId}/rotate with the ?targetAlgorithm= query param appended. Platform validates the algorithm against the configured primitive’s SupportsAlgorithm registry before launching the walker; an unsupported algorithm returns HTTP 400 (unsupported_target_algorithm / primitive_unavailable) without mutating state.
  4. Streams walker progress to stdout, identical shape to stella crypto kek rotate --confirm.

Exit codes (from CryptoCommandGroup + StreamRotationProgressAsync + the typed KEK client’s HTTP→exit-code map):

2. Override the KEK id if your install uses a non-default

The CLI defaults to --reseal-kek-id platform:connector-credentials:v1. Override if your install uses a different KEK id:

stella crypto profile set sm --reseal \
  --tenant <tenant-id> \
  --reseal-kek-id platform:connector-credentials:v2

Monitor progress

The CLI streams per-batch progress. The same data is available through:

Mixed-state semantics (expected; not a bug)

During the walker run, some rows on this kek_id store will still be on the prior algorithm/version (the walker advances batch by batch). This is expected and safe:

If your install has a high credential-access rate, some rows may re-seal naturally via Sprint 027’s “Seal-on-update” path before the walker reaches them — this is fine; the walker’s atomic swap ignores rows that are already on the target algorithm.

Verification post-switch

# 1. Profile is now persisted. `profile show` prints a fixed text table
#    (no --format json option, no algorithm field):
stella crypto profile show --tenant <tenant-id>
# Expected (text):
#   Tenant cryptography
#     Tenant:     <tenant-id>
#     Profile:    sm
#     Updated by: <actor>

# 2. Validate returns ok under the new profile. (Validates the active
#    request tenant — no --tenant arg; see Prerequisites step 3.)
stella crypto profile validate --format json
# Expected: { "outcome": "ok", "activeAlgorithm": "SM4-GCM", ... }

# 3. (Roadmap) The per-algorithm UI chip only reflects the switch once the
#    Platform server emits populations.algorithmBreakdown. Until then, the
#    list page /console-admin/crypto-control (Blob Populations panel) shows
#    per-store version counts, not a per-algorithm breakdown. (The per-kek
#    detail page /console-admin/crypto-control/<kekId> shows active version +
#    rotation history only — no population bar.)

# 4. Scriptable check — per-store version populations (no per-tenant /
#    per-algorithm field exists in this payload).
stella crypto kek status --kek-id <id> --format json | \
  jq '.populations.stores'
# Confirm the switch authoritatively via the per-row algorithm column,
# e.g. SELECT secret_alg, count(*) FROM platform.connector_credentials
#      WHERE tenant_id = '<tenant-id>' GROUP BY secret_alg;

Rollback

The walker is forward-only in mechanics (each row is re-sealed in place under the new algorithm) but bidirectional in workflow:

# Switch back to the prior posture.
stella crypto profile set world --reseal --tenant <tenant-id>

This sets the tenant’s profile back to world and runs the kek_id-wide walker again; the named tenant’s rows re-seal to AES-256-GCM (other tenants’ rows are re-sealed under their own profile’s algorithm, as in any pass). The walker treats the prior algorithm as the source and the new algorithm as the target; there is nothing special about the direction.

Caveat: the regional plugin must remain loaded for as long as ANY row on this install (across all tenants) carries the regional algorithm. If you unload the regional plugin while rows still reference it via their per-row algorithm column (secret_alg / credential_alg / alg per store), Open on those rows returns null rather than silently falling back to AES — the credential primitives are contractually non-throwing on Open (no plaintext-bearing exceptions), so the plugin-routed primitive and CredentialAead.Open return null for an unresolved algorithm and the caller surfaces it as a credential-decrypt failure (e.g. the resolve-credentials path returns a failed result / audit entry). There is no silent downgrade to AES — that would be a compliance-violation gate.

Abort

Identical to KEK rotation abort:

stella crypto kek abort --kek-id <id> --rotation-id <id>

Rows already re-sealed under the new algorithm remain there; rows still on the prior algorithm stay on the prior algorithm. The mixed-state is safe (per the mixed-state section above). Plan a follow-up rotation if needed.

Concurrent rotations

Audit & sentinel guarantees