Secret Detection Bundle Rotation

Audience: Scanner operators, Security Guild, Offline Kit maintainers.

Related: secret-leak-detection.md

1. Overview

Secret detection rule bundles are versioned, immutable, signed artifacts that define the patterns Stella Ops uses to detect leaked credentials during a scan. This runbook covers the full operational lifecycle: the versioning strategy, how to roll a new bundle out (online and air-gapped), how to roll back, and how to monitor a rotation for regressions.

If you are looking for what the rules detect rather than how to ship them, start with secret-leak-detection.md.

2. Versioning strategy

Bundles follow CalVer (Calendar Versioning) with the format YYYY.MM:

Version precedence:

• 2026.02 > 2026.01.1 > 2026.01
• Patch versions (YYYY.MM.N) are only used for critical fixes
• Monthly releases reset the patch counter

Custom bundles: Organizations creating custom bundles should use a prefix to avoid conflicts:

• myorg.2026.01 for organization-specific bundles
• Or semantic versioning: 1.0.0, 1.1.0, etc.

3. Release cadence

4. Rotation procedures

4.1 Downloading the new bundle

# From the Export Center or Offline Kit
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.manifest.json
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.rules.jsonl
curl -O https://export.stellaops.io/rules/secrets/2026.02/secrets.ruleset.dsse.json

For air-gapped environments, obtain the bundle from the Offline Kit media.

4.2 Verifying the bundle

Always verify bundles before deployment:

stella secrets bundle verify \
  --bundle ./2026.02 \
  --shared-secret-file /etc/stellaops/secrets-signing.key \
  --trusted-key-ids stellaops-secrets-signer

Expected output:

Bundle verified successfully.
  Bundle ID:      secrets.ruleset
  Version:        2026.02
  Rule count:     18
  Enabled rules:  18
  Signed by:      stellaops-secrets-signer
  Signed at:      2026-02-01T00:00:00Z

4.3 Staged rollout

For production environments, use a staged rollout:

Stage 1: Canary (1 worker)

# Deploy to canary worker
scp -r ./2026.02/* canary-worker:/opt/stellaops/plugins/scanner/analyzers/secrets/
ssh canary-worker 'systemctl restart stellaops-scanner-worker'

# Monitor for 24 hours
# Check logs, metrics, and finding counts

Stage 2: Ring 1 (10% of workers)

# Deploy to ring 1 workers
ansible-playbook -l ring1 deploy-secrets-bundle.yml -e bundle_version=2026.02

Stage 3: Full rollout (all workers)

# Deploy to all workers
ansible-playbook deploy-secrets-bundle.yml -e bundle_version=2026.02

4.4 Atomic deployment

For single-worker deployments or when downtime is acceptable:

# Stop the worker
systemctl stop stellaops-scanner-worker

# Backup current bundle
cp -r /opt/stellaops/plugins/scanner/analyzers/secrets{,.backup}

# Deploy new bundle
cp -r ./2026.02/* /opt/stellaops/plugins/scanner/analyzers/secrets/

# Start the worker
systemctl start stellaops-scanner-worker

# Verify startup
journalctl -u stellaops-scanner-worker | grep SecretsAnalyzerHost

4.5 Using symlinks (recommended)

For zero-downtime rotations, use the symlink pattern:

# Directory structure
/opt/stellaops/plugins/scanner/analyzers/secrets/
  bundles/
    2026.01/
      secrets.ruleset.manifest.json
      secrets.ruleset.rules.jsonl
      secrets.ruleset.dsse.json
    2026.02/
      secrets.ruleset.manifest.json
      secrets.ruleset.rules.jsonl
      secrets.ruleset.dsse.json
  current -> bundles/2026.02  # Symlink

Rotation with symlinks:

# Deploy new bundle (no restart needed yet)
cp -r ./2026.02 /opt/stellaops/plugins/scanner/analyzers/secrets/bundles/

# Atomic switch
ln -sfn bundles/2026.02 /opt/stellaops/plugins/scanner/analyzers/secrets/current

# Restart worker to pick up new bundle
systemctl restart stellaops-scanner-worker

5. Rollback procedures

5.1 Quick rollback

If issues are detected after deployment:

# With symlinks (fastest)
ln -sfn bundles/2026.01 /opt/stellaops/plugins/scanner/analyzers/secrets/current
systemctl restart stellaops-scanner-worker

# Without symlinks
cp -r /opt/stellaops/plugins/scanner/analyzers/secrets.backup/* \
      /opt/stellaops/plugins/scanner/analyzers/secrets/
systemctl restart stellaops-scanner-worker

5.2 Identifying rollback triggers

Roll back immediately if you observe:

5.3 Post-rollback verification

After rolling back:

# Verify worker is healthy
systemctl status stellaops-scanner-worker

# Check bundle version in logs
journalctl -u stellaops-scanner-worker | grep "Loaded bundle"

# Verify finding generation (run a test scan)
stella scan --target test-image:latest --secrets-only

6. Bundle retention

Retain previous bundle versions for rollback capability:

Cleanup script:

#!/bin/bash
BUNDLE_DIR=/opt/stellaops/plugins/scanner/analyzers/secrets/bundles
KEEP=3

ls -dt ${BUNDLE_DIR}/*/ | tail -n +$((KEEP+1)) | xargs rm -rf

7. Monitoring rotation

Key metrics to monitor during rotation:

Prometheus alert example:

- alert: SecretBundleRotationAnomaly
  expr: |
    abs(
      sum(rate(scanner_secret_finding_total[5m]))
      - sum(rate(scanner_secret_finding_total[5m] offset 1h))
    ) / sum(rate(scanner_secret_finding_total[5m] offset 1h)) > 0.5
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Secret finding rate changed significantly after bundle rotation"

8. Air-gapped rotation

For air-gapped environments the recommended flow uses the new stella scanner secrets bundle CLI (introduced in Sprint 20260512_038) which combines verification + atomic swap and emits a metadata-only invalidation notice on the shared Valkey channel scanner-secrets-bundle-changed so worker processes drop their stale ruleset without waiting on the cache TTL.

Bundle assembly. The signed secret bundles distributed via this flow can be packaged together with the wider in-tree airgap artefacts using the baseline Offline Kit assembler — see offline-kit-assembly.md. The assembler auto-includes __Datasets/secrets/** if present, alongside the trust-store seeds and signed plug-in manifests, and emits a hash-verifiable manifest the receiving operator can audit before importing with stella scanner secrets bundle import.

Bundle rotation cadence. The Offline Kit ships weekly via the Gitea Actions lane .gitea/workflows/offline-kit-assembly.yml (Sprint 20260513_018). Operational discipline for the offline-kit lane — when to bump, key custody, publication path, consumer verification — is captured in the dedicated rotation runbook: offline-kit-rotation.md. Air-gap sites that rotate secret bundles typically rotate the Offline Kit at the same time, since both ride the same physical-media or mirror channel.

1. Obtain bundle from secure media:

   # Mount offline kit media
   mount /dev/sr0 /mnt/offline-kit
   
   # Stage onto the server-local filesystem
   cp -r /mnt/offline-kit/rules/secrets/2026.02 /var/tmp/incoming-bundle/
   

2. Verify with the local shared secret (no network egress):

   stella scanner secrets bundle verify \
     --bundle /var/tmp/incoming-bundle \
     --secret-file /etc/stellaops/offline-signing.key \
     --trusted-key-id stellaops-secrets-signer
   

3. Import the bundle (verifies + atomic-swaps the active version):

   stella scanner secrets bundle import \
     --bundle /var/tmp/incoming-bundle \
     --store /opt/stellaops/scanner/secrets/store \
     --secret-file /etc/stellaops/offline-signing.key \
     --trusted-key-id stellaops-secrets-signer
   

   The CLI writes the new bundle under <store>/bundles/<version>/, rewrites the <store>/current.txt pointer via an atomic rename, and publishes a metadata-only change notice on the scanner-secrets-bundle-changed Valkey channel. Worker processes subscribed to that channel reload their ruleset before the next scan.

4. Confirm activation:

   stella scanner secrets bundle list \
     --store /opt/stellaops/scanner/secrets/store --format json
   

   Look for "active": true next to the version you just imported.

8.1 Server-side import (operator API)

When the operator workstation can reach the Scanner WebService on the air-gap side, the same flow can be triggered via:

POST /api/v1/scanner/secrets/bundles
Authorization: Bearer <token with secrets:bundle:write scope>
Content-Type: application/json

{
  "bundlePath": "/var/tmp/incoming-bundle",
  "sharedSecretFile": "/etc/stellaops/offline-signing.key",
  "trustedKeyIds": ["stellaops-secrets-signer"]
}

The endpoint enforces the secrets:bundle:write scope (scanner.secrets.bundle.write policy) and re-runs the same BundleVerifier the CLI uses, so the verification posture is identical across CLI and API. A successful response carries the staged path and the new manifest SHA-256 stamp for the operator’s audit log.

GET /api/v1/scanner/secrets/bundles (scope secrets:bundle:read) returns the list of staged versions and highlights the active one.

8.2 Rotation cadence in air-gap mode

The Valkey notification carries only { bundleId, version, action, stampedAt } — no rule patterns, no signing material, no plaintext findings. This mirrors the discipline established for the connector-credentials-changed channel (RFC-0001 §7).

9. Emergency procedures

9.1 Disabling secret detection

If secret detection must be disabled entirely:

# Disable via configuration
echo 'scanner.features.experimental.secret-leak-detection: false' >> /etc/stellaops/scanner.yaml

# Restart worker
systemctl restart stellaops-scanner-worker

9.2 Emergency rule disable

To disable a specific problematic rule without full rotation:

1. Edit the manifest to set enabled: false for the rule
2. This breaks signature verification (expected)
3. Configure worker to skip signature verification temporarily:

   scanner:
     secrets:
       skipSignatureVerification: true  # TEMPORARY - re-enable after fix
   

4. Restart worker
5. Request emergency patch release from Security Guild

10. References

• secret-leak-detection.md - Main secret detection documentation
• SPRINT_20260104_003_SCANNER_secret_rule_bundles.md - Implementation sprint
• dsse-rekor-operator-guide.md - DSSE and Rekor verification
• offline-kit-assembly.md - Baseline Offline Kit assembler (Sprint 20260513_011) — packages signed secret bundles + trust-store + plug-in manifests into a deterministic tarball.
• offline-kit-rotation.md - Offline Kit rotation runbook (Sprint 20260513_018) — cadence (weekly Mondays 04:00 UTC), key-custody policy (dev key vs production HSM-backed), publication paths (Gitea release / mirror / physical media), and consumer verification flow. Sibling discipline to this doc — air-gap sites that rotate secret bundles typically rotate Offline Kits at the same time.