Development key hierarchy — one root, derived purposes

Status: implemented 2026-08-11 (sprint SPRINT_20260803_001 KEY-3, owner ruling B3). Code: src/__Libraries/StellaOps.Cryptography/DevKeys/. Binding rule this serves: AGENTS.md §2.12 — no committed private key material, keys resolve from runtime sources, consumers fail closed when a key is absent.

The problem this solves

§2.12 says a development lab may not ship committed keys, and that every consuming service must fail closed when its key is missing. Taken alone, that turns bringing up a lab into a chore: four services, four openssl invocations, four paths to remember, four things to redo after a volume reset. The predictable outcome is that somebody commits a key again “just for dev”.

The owner’s ruling (B3, 2026-08-03) is the way out:

“perhaps there should be dev / normal setup just for these. i also would like to have one key generate and derivates of it for the other cases. derivates should be automatic, not static”

So: one generated secret per environment, and every per-purpose key computed from it on demand. Nothing is stored but the root. Rotating the root rotates everything.

The trust argument — read this before using it anywhere real

This is a convenience inside one lab trust domain. It is not a production custody model, and it is not a key-management system.

The reason is structural, not a matter of hardening: every derived key is a pure function of the root, so the root is equivalent to all of them at once. Whoever holds it can reconstruct the Authority signing key, the ack-token key, the scanner attestation key and the registry token key without touching any of those services. That is a fine property when the whole point is “one command brings up my laptop’s lab”, and an unacceptable one when the keys are supposed to have independent blast radii, separate custody, or per-key rotation — which is exactly what a production estate needs.

Concretely, in production each key is supplied independently by the operator (Vault or a mounted file) and rotates on its own schedule. The dev hierarchy is not a weaker version of that. It is a different thing, kept behind a gate so the two cannot be confused.

Dev vs normal — how a service chooses

Every key-consuming service resolves in this fixed order:

1. The operator’s key file / Vault secret. Always first, always wins. Once a key file is configured, it is authoritative: if it cannot be resolved, the service throws rather than falling back to anything weaker. (This is the NIS2 SoA rule from §2.12 — a stale mount must not silently downgrade custody.)
2. The derived dev key — only if STELLAOPS_DEV_KEYS_ENABLED=true.
3. Fail closed, with an error naming the config key, the resolved path, and the openssl command that would produce the key.

Step 2 is unreachable without the gate, and the gate accepts only the exact string true. 1, yes, on, truthy and an empty value all mean off — pinned by Derivation_is_disabled_unless_the_gate_is_explicitly_true, which asserts every one of them. A production host that has never heard of this feature therefore cannot reach the derived path, which is the property the ruling asked for (“impossible to enable accidentally in production”).

Setting it up

# One secret per environment. devops/etc/secrets/ is already git-ignored.
openssl rand -base64 32 > devops/etc/secrets/dev-key-root.b64
chmod 600 devops/etc/secrets/dev-key-root.b64

export STELLAOPS_DEV_KEYS_ENABLED=true
export STELLAOPS_DEV_KEY_ROOT_FILE=devops/etc/secrets/dev-key-root.b64

STELLAOPS_DEV_KEY_ROOT carries the base64 secret inline instead, for throwaway containers. When both are set the file wins and the inline value is ignored entirely.

Rotating is one command: overwrite the root file. Every purpose changes on the next start; nothing else needs touching. Tokens minted under the old keys stop validating, so consumers must refetch JWKS — the same consequence as any signing-key rotation.

Derivation

No ad-hoc construction is introduced. That was a condition the sprint placed on derived asymmetric keys, and it is why each row above names a published standard.

Why rejection sampling and not candidate mod n. Reduction is biased toward small scalars, because the byte-aligned candidate range is not a whole multiple of the group order. The bias is small, but it is a real weakening of a signing key and avoiding it costs one loop. P-521 gets its 7 surplus high bits masked first, or ~99% of candidates would be rejected.

Two caveats on the RSA path, both real and both the reason to prefer an EC purpose wherever the consumer permits one:

• Reproducibility is bounded by BouncyCastle. The key is stable for a given root, label and library version — an upstream generator change can change the key. Acceptable for a dev convenience; it would not be acceptable for anything durable.
• It is slow. Prime search dominates: ~1.2 s per 3072-bit derivation against microseconds for EC. DevKeyPurpose.RegistryTokenSigning is RSA only because that service is hard-wired to SecurityAlgorithms.RsaSha256 and reads an RSA private key off the certificate (SigningKeyLoader).

Purposes

Labels are the HKDF info string, so they are contract: renaming one silently rotates that consumer’s key, and duplicating one silently fuses two purposes into a shared key. Add labels freely; never repurpose an existing one. DevKeyHierarchyTests pins uniqueness and the stellaops/dev/<purpose>/v1 shape against exactly those two mistakes.

The registry-token derived key is cached for the process, keyed by DevKeyRoot.Fingerprint (lowercase hex SHA-256 of the root — a non-secret identifier, also used as the token kid prefix so a header distinguishes a lab signature from a custody-held one). That cache is a correctness concern, not only a performance one: an RSA derivation replays a prime search, and a cache keyed on anything but the root’s identity would serve one root’s key to another.

The lab’s operator key files (2026-08-21)

The derived hierarchy above is the convenience path. The lab itself runs on explicit operator key files, because §2.12 precedence says an operator’s file always beats anything derived or inline — the derived key is only reachable once File.Exists has already failed. All three live at the git-ignored devops/etc/secrets/ and none of them can be committed:

The attestor having its own key is the point, not an accident of naming. Until 2026-08-21 the attestor signed with Authority’s signing key — the same file on disk, reached through a second bind mount (devops/etc/authority/keys → /app/etc/certs). Nothing declared that; it was a property of the mount, which is why a sweep for Authority’s in-container path could never have found it. One key covering both meant that a single compromised file forged both platform tokens and the evidence the estate’s own verifier accepts — the two things the product’s central claim rests on, sharing one blast radius.

This also brings the lab into line with what the installers already did. install.sh:260-268 and install.ps1:372-380 have always generated two distinct EC keys and injected the attestor’s matching public half through a runtime-only compose overlay, so an installed estate never had the collision. The lab was the outlier. If you are reasoning about key custody, read the installers as the intended topology and the lab as something that has to be kept matching it.

Rotating the attestor key means rewriting two PUBLIC pins, not one. Both are inline literals in compose and both are fail-closed, so a rotation that moves only the private key leaves the attestor signing with material its own trust list rejects:

• ATTESTOR_ATTESTOR__VERDICTTRUST__KEYS__0__PUBLICKEYPEM — the trusted verdict roster.
• ATTESTOR_ATTESTOR__REKOR__PRIMARY__PUBLICKEYBASE64 — DER SPKI, base64, used for local transparency checkpoint verification. Two consecutive consumer sweeps missed this one.

openssl pkey -in devops/etc/secrets/attestor-signing.pem -pubout                  # the PEM pin
openssl pkey -in devops/etc/secrets/attestor-signing.pem -pubout -outform DER | base64 -w0   # the base64 pin

The general rule: sweep for the key’s public half BY VALUE, not just for its filename. A filename grep cannot find an inline literal, and a private-path grep cannot find a public one.

Signing a plugin bundle offline (stella plugin sign)

The CLI could not sign at all before KEY-3 — it composes exactly one signing provider and nothing registered a key into it from the plugin path, so --key-id always named a key that was not there. --key-file is now the operator source and takes precedence over everything; the derived key is the lab fallback.

# Operator path — the one to use for anything real.
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out plugin-signing.pem
chmod 600 plugin-signing.pem
stella plugin sign ./my-plugin \
  --assembly MyPlugin.dll --key-id my-signing-root-1 \
  --profile world --algorithm ES256 \
  --key-file plugin-signing.pem

# Development lab — no key file; derived from your own root. Prints a warning naming the root.
export STELLAOPS_DEV_KEYS_ENABLED=true
export STELLAOPS_DEV_KEY_ROOT_FILE=devops/etc/secrets/dev-key-root.b64
stella plugin sign ./my-plugin \
  --assembly MyPlugin.dll --key-id my-signing-root-1 \
  --profile world --algorithm ES256

The bundle directory name must equal the plugin id. Both forms write <assembly>.sig and a runtime-bundle.v2 manifest, and PluginManifestSigner self-verifies before writing, so a signer that cannot verify its own output fails instead of emitting an unusable bundle.

stella plugin release takes the same two options and the same key source. Release signs through the same PluginManifestSigner, so it would otherwise fail exactly as sign used to:

stella plugin release ./my-plugin --assembly MyPlugin.dll --key-id my-signing-root-1 \
  --profile world --algorithm ES256 --key-file plugin-signing.pem \
  --registry <host>/stella-ops/plugins/my-plugin --no-register

Omit --key-file and set the dev-gate variables for the lab form. The derived-key warning is emitted on this path too, and it matters more here — release publishes the artifact.

--algorithm ES256 is required today, and that is a defect elsewhere, not a quirk of this feature. PluginSigningProfileAlgorithms maps world and fips to RS256, and no ICryptoProvider in the estate implements RS256 signing — DefaultCryptoProvider advertises ES256 only and rejects non-EC keys outright (UpsertSigningKey → crypto.provider.ec_keys_only). So those two profiles are structurally unsignable for every host, not just the CLI, until either the provider gains RSA or the profile map changes. The verifier is unaffected: it resolves by (algorithm, keyId) from the manifest entry and does not re-derive the algorithm from the profile, so an ES256-signed world bundle verifies normally. Recorded for the owner in SPRINT_20260803_001 — KEY-3 deliberately did not widen a shared crypto library to work around it.

Two failure modes now name themselves rather than sharing one opaque crypto.registry.signing_not_supported (the CLI never initialises the localization registry, so that resource key is what an operator actually saw):

What the library deliberately does not do

• It never writes to disk. DevKeyRoot.Generate() returns a secret and leaves persistence to the caller. A library that silently generates and stores key material is the failure mode §2.12 exists to prevent — the operator must know where their root lives.
• It never invents a root. Gate on with no root supplied is an error with generation guidance, not an opportunity to make one up. An operator who asked for dev keys and gave no root made a mistake worth naming.
• It has no production mode. There is no flag that turns this into operator custody. Production keys come from Vault or operator files, through each service’s own loader.

Related

• AGENTS.md §2.12 — the committed-key-material rulebook.
• tools/scripts/validate/check-key-material.sh — the tree-wide sweep, plus --self-test.
• docs/implplan/SPRINT_20260803_001_Cryptography_dev_key_hierarchy.md — KEY-1…KEY-4, including the credential inventory this hierarchy is meant to shrink.
• Authority architecture §6 — why a key rotation must rotate the kid with it.
• Authority / gateway auth troubleshooting §5 — the post-rotation 401 symptom and how to tell it from a real fault.