Offline Kit — rotation runbook

Audience: Scanner operators, Security Guild, Offline Kit maintainers, air-gap site administrators.

Companion docs: offline-kit-assembly.md(the assembler contract this runbook drives) and secrets-bundle-rotation.md(sibling rotation discipline for secret-detection rule bundles).

This runbook covers the productionised Offline Kit baseline pipeline: when to rotate, who custodies the signing key, how to publish the new release, and how downstream consumers verify what they received.

The baseline pipeline itself (the deterministic assembler + manifest contract) is documented in offline-kit-assembly.md. This document sits on top of that contract and adds the operational discipline.

1. When to rotate

The weekly cadence keeps the Offline Kit fresh against the dataset PRs that land in main between releases while bounding the artifact retention budget. Consumers should treat any kit older than 30 days as expired and re-pull from the latest scheduled run.

1.1 Skipping a scheduled run

A scheduled run is never skipped silently. If the assembler fails (e.g., determinism harness drift, dataset enumeration error), the workflow fails loudly and a ::error:: annotation is recorded. The operator on call must then:

1. Inspect the failed run’s logs in the Gitea Actions UI.
2. If the cause is a transient runner issue, re-trigger via workflow_dispatch.
3. If the cause is a determinism regression, STOP — file a Sprint ticket against the assembler. Do NOT publish a drifted kit.
4. If the cause is a dataset / manifest issue, fix the upstream change and let the next scheduled run pick it up.

2. How to publish a new release

The CI workflow uploads two/three artifacts under the name offline-kit-<version>-<run-id>:

offline-kit-baseline-<version>.tar.gz             # The kit itself.
offline-kit-baseline-<version>.manifest.json      # Per-file sha256 manifest.
offline-kit-baseline-<version>.summary.json       # Run summary (CI-emitted).
offline-kit-baseline-<version>.manifest.signed.json  # Only when signing is enabled — see §3.

2.1 Retrieving the artifact from a Gitea Actions run

1. Open the latest successful scheduled run of the Offline Kit Assembly workflow.
2. Scroll to Artifacts and download offline-kit-<version>-<run-id>.
3. Verify the bundle’s sha256 matches the archive.sha256 field in summary.json before publication.

sha256sum offline-kit-baseline-2026.05.13.tar.gz
# Compare against:
jq -r '.archive.sha256' offline-kit-baseline-2026.05.13.summary.json

2.2 Publishing to operators

There are three supported distribution paths, in order of preference:

1. Gitea release attachment. Attach the .tar.gz + .manifest.json
   • .summary.json (and .manifest.signed.json when present) to a Gitea release tagged offline-kit/<version>. This is the recommended path because Gitea releases carry release notes + an immutable URL.
2. Internal artifact mirror. Push to an internal HTTPS-only mirror the operator already trusts. The mirror MUST publish the per-file sha256 manifest alongside the archive.
3. Physical media (air-gap) — for sites without operator-facing network egress. Burn the archive + manifest to optical media. Carry the summary.json separately (e.g., printed page) so the receiving operator has an out-of-band channel for the sha256 they can trust.

Cross-reference: the parallel release-manifest-verify-publish.yml lane handles the signed release-manifest publication for the full release artifacts (SBOMs, DSSE envelopes, OCI referrers). The offline-kit baseline lane documented here is the lightweight slice of the in-tree air-gap artifacts only.

2.3 Versioning convention

The assembler accepts any string matching [A-Za-z0-9._-]+. The CI workflow validates this regex before invoking the assembler.

3. Key custody policy

The Offline Kit baseline manifest can optionally carry a DSSE-style signature so consumers can verify provenance independent of the distribution channel. Two key custody lanes are defined:

3.1 Dev key (CI dry-run only)

• Location: src/__Tests/__Datasets/trust-store/scanner-plugins/dev/seed.hex (committed to the repo on purpose — see the README in that directory).
• Use case: smoke-testing the signing pipeline in CI. The signature is produced and uploaded with the artifact under the mode: "ci-dry-run" marker in summary.json.
• NOT FOR PRODUCTION. A bundle signed with this key is recognisable by its key id (stella-scanner-plugin-dev-…) and the summary.signing.mode == "ci-dry-run" field. Air-gap site operators must reject any kit whose summary.signing.mode != "production".

3.2 Production key (HSM-backed, two-person)

The production signing key is not yet provisioned. When ops adopts a custody policy, the following invariants apply:

• Key form: Ed25519 seed (32 bytes, RFC 8032 §5.1.5) — same format the dev key uses, same format the Scanner.Plugin.SigningTool sign --key <path> flag accepts.
• Storage: HSM-backed (preferred) or sealed file under two-person physical custody. NEVER on a shared filesystem.
• CI access: the seed is exposed to the Gitea Actions runner as the secret OFFLINE_KIT_SIGNING_KEY. The workflow writes it to RUNNER_TEMP with umask 077, signs, then shred -us the file.
• Two-person rotation: any rotation of the production key requires sign-off by two named custodians. The custody log lives at docs/security/key-rotation-log.md (TODO — file lives there once the first production key is provisioned).
• Audit log: the Gitea Actions run id and the new key id are appended to the custody log on each rotation.
• Trust-anchor distribution: the production public key (and any previous key still in the rotation window) is shipped to consumers out-of-band via the Gitea release page and the operator-facing documentation. Consumers must pin a trust-store snapshot and refuse signatures from unknown key ids.

Open decision (out of scope for Sprint 20260513_018): custody policy specifics (HSM vendor, two-person ceremony, rotation frequency). This doc fixes the contract; the operations team owns the policy. Until ops chooses, the OFFLINE_KIT_SIGNING_KEY secret remains unset and the workflow emits unsigned kits.

3.3 Signing roadmap

The Sprint 018 workflow has the signing infrastructure wired as a seam (setup-dotnet, build, key-staging, shred) but the actual signing call fails by design when invoked, because the current Scanner.Plugin.SigningTool validates its input as a ScannerPluginManifest (a different shape than the offline-kit baseline manifest). The future sprint that closes this gap will:

1. Add a kit-shaped DSSE signer that uses the same Ed25519 primitives.
2. Flip the workflow’s default enable_signing input from false to true.
3. Replace the “Sign manifest (PLACEHOLDER)” step body with the real signer invocation.

Until that sprint lands, Offline Kit bundles ship unsigned and consumers verify via the per-file sha256 manifest exclusively (see §4).

4. How consumers verify

Three verification layers are available; each is independently useful.

4.1 Archive sha256

The simplest check: the operator who downloaded the kit confirms its sha256 against the value quoted in the Gitea release page, the summary.json file, or an out-of-band channel.

sha256sum offline-kit-baseline-2026.05.13.tar.gz

Expected output matches the archive.sha256 field from the run’s summary.json.

4.2 Per-file manifest

Once extracted, the receiving operator can re-hash every file in the archive and compare against the in-archive manifest (0-manifest.json). The bash recipe is in offline-kit-assembly.md§6 and reproduced here for clarity:

mkdir -p /tmp/kit && tar -xzf offline-kit-baseline-2026.05.13.tar.gz -C /tmp/kit
cd /tmp/kit

# Spot-check
sha256sum datasets/trust-store/scanner-plugins/dev/index.json
# -> compare with "sha256" entry in 0-manifest.json for the same path

# Full pass
python3 - <<'PY'
import hashlib, json, pathlib, sys
m = json.load(open('0-manifest.json'))
bad = 0
for entry in m['files']:
    p = pathlib.Path(entry['path'])
    h = hashlib.sha256(p.read_bytes()).hexdigest()
    if h != entry['sha256']:
        print(f'MISMATCH {p}: got {h} expected {entry["sha256"]}', file=sys.stderr)
        bad += 1
print(f'{len(m["files"])} files; {bad} mismatches')
sys.exit(1 if bad else 0)
PY

This layer alone is sufficient for an air-gap operator who trusts the distribution channel (e.g., physical media + an out-of-band sha256 quote).

4.3 Signed manifest (future, see §3.3)

When the kit-shaped DSSE signer lands, the receiving operator will run:

dotnet run --project src/Scanner/__Libraries/StellaOps.Scanner.Plugin.SigningTool \
  -- verify \
  --manifest offline-kit-baseline-2026.05.13.manifest.signed.json \
  --trust-store /etc/stellaops/offline-kit/trust-store

The trust-store is the operator’s pinned snapshot of the production public key(s) (§3.2). Until the kit-shaped signer lands, this verb fails on offline-kit manifests by design — consumers must rely on §4.1 + §4.2.

5. Failure modes and incident response

6. Cross-references

• offline-kit-assembly.md— the assembler contract this runbook drives.
• secrets-bundle-rotation.md— sibling rotation discipline; the secret-bundle distribution path piggy-backs on this Offline Kit flow when air-gap operators take both at once.
• SPRINT_20260513_018_DevOps_offline_kit_ci_workflow_rotation.md — the sprint that productionised this lane.
• SPRINT_20260513_011_DevOps_offline_kit_assembly_baseline.md — the prior baseline sprint that landed the assembler + determinism harness.
• .gitea/workflows/offline-kit-assembly.yml — the CI workflow.
• .gitea/workflows/release-manifest-verify-publish.yml — the parallel release-manifest lane (full release artifacts, not the baseline).