Authority Signing Key Rotation Playbook

Audience: operators and on-call engineers rotating the Stella Ops Authority signing keys.

This playbook describes the repeatable, zero-downtime sequence for rotating the ES256 keys that the Authority service uses to sign JWKS and revocation bundles. Use it together with the Authority service guide and the automation shipped under ops/authority/.

1. Overview

Authority publishes JWKS and revocation bundles signed with ES256 keys. To rotate those keys without downtime, Stella Ops ships:

• Automation script: ops/authority/key-rotation.sh
  Shell helper that POSTS to /internal/signing/rotate, supports metadata, dry-run, and confirms JWKS afterwards.
• CI workflow: .gitea/workflows/authority-key-rotation.yml
  Manual-dispatch workflow that pulls environment-specific secrets, runs the script, and records the result. It works across staging and production by passing the environment input.

2. Prerequisites

Production Authority startup now binds OpenIddict JWT signing to the configured Authority signing key. Non-local environments must provide a stable signing.activeKeyId, a key location or provider handle in signing.keyPath, and a signing.keySource/signing.provider pair that resolves through the Authority signing source and crypto-provider registry before the service starts; Development/Testing are the only runtimes that may use ephemeral OpenIddict signing keys.

1. Generate a new PEM key (per environment)

   openssl ecparam -name prime256v1 -genkey -noout \
     -out certificates/authority-signing-<env>-<year>.pem
   chmod 600 certificates/authority-signing-<env>-<year>.pem
   

2. Stash the previous key under the same volume so it can be referenced in signing.additionalKeys after rotation.
3. Ensure secrets/vars exist in Gitea
   • <ENV>_AUTHORITY_BOOTSTRAP_KEY
   • <ENV>_AUTHORITY_URL
   • Optional shared defaults AUTHORITY_BOOTSTRAP_KEY, AUTHORITY_URL.

3. Executing the rotation

Option A – via CI workflow (recommended)

1. Navigate to Actions → Authority Key Rotation.
2. Provide inputs:
   • environment: staging, production, etc.
   • key_id: new kid (e.g. authority-signing-2025-dev).
   • key_path: path as seen by the Authority service (e.g. ../certificates/authority-signing-2025-dev.pem).
   • Optional metadata: comma-separated key=value pairs (for audit trails).
3. Trigger. The workflow:
   • Reads the bootstrap key/URL from secrets.
   • Runs ops/authority/key-rotation.sh.
   • Prints the JWKS response for verification.

Option B – manual shell invocation

AUTHORITY_BOOTSTRAP_KEY=$(cat /secure/authority-bootstrap.key) \
./ops/authority/key-rotation.sh \
  --authority-url https://authority.example.com \
  --key-id authority-signing-2025-dev \
  --key-path ../certificates/authority-signing-2025-dev.pem \
  --meta rotatedBy=ops --meta changeTicket=OPS-1234

Use --dry-run to inspect the payload before execution.

4. Post-rotation checklist

1. Update authority.yaml (or environment-specific overrides):
   • Set signing.activeKeyId to the new key.
   • Set signing.keyPath to the new PEM.
   • Append the previous key into signing.additionalKeys.
   • Ensure keySource/provider match the values passed to the script.
2. Run stellaops-cli auth revoke export so revocation bundles are re-signed with the new key.
3. Confirm /jwks lists the new kid with status: "active" and the previous one as retired.
4. Confirm tokens issued after restart carry the active kid and validate against the live /jwks response.
5. Archive the old key securely; keep it available until all tokens/bundles signed with it have expired.

Provider-backed keys: KMS/HSM/regional providers may issue OpenIddict JWTs when the configured signer returns JWS-compatible signatures and exports a public JWK whose kid and alg match the active Authority signing key. Startup fails closed if the provider cannot resolve the signer or cannot export the verification key used by /jwks.

5. Development key state

For the sample configuration (etc/authority.yaml.sample) we minted a placeholder dev key:

• Active: authority-signing-2025-dev (certificates/authority-signing-2025-dev.pem)
• Retired: authority-signing-dev

Treat these as examples; real environments must maintain their own PEM material.

6. References

• Authority service guide – Architecture and rotation SOP (Section 5).
• Backup & restore – Recovery flow that references this playbook.
• Monitoring & alerting – Telemetry to watch during and after a rotation.
• ops/authority/README.md – Rotation-script usage and examples.
• scripts/rotate-policy-cli-secret.sh – Helper to mint new policy-cli shared secrets when policy scope bundles change.

7. Appendix — Policy CLI secret rotation

Scope migrations such as AUTH-POLICY-23-004 require issuing fresh credentials for the policy-cli client. Use the helper script committed with the repo to keep secrets deterministic across environments.

./scripts/rotate-policy-cli-secret.sh --output etc/secrets/policy-cli.secret

The script writes a timestamped header and a random secret into the target file. Use --dry-run when generating material for external secret stores. After updating secrets in staging/production, recycle the Authority pods and confirm the new client credentials work before the next release freeze.