Stella Ops Disaster Recovery Guide

Audience: TUF/trust administrators, security responders, and platform operators. Purpose: Recover the Stella Ops trust infrastructure during a Rekor outage, key compromise, or TUF repository failure, using the shipped stella trust CLI and devops/scripts/ tooling.

Sprint: SPRINT_20260125_003 — WORKFLOW-003. Last updated: 2026-05-30 (reconciled against src/Attestor, src/Cli, and devops/scripts/).

Tooling note (verified against source): The DR scripts referenced below (disaster-swap-endpoint.sh, rotate-rekor-key.sh, rotate-signing-key.sh, bootstrap-trust-offline.sh) exist under devops/scripts/. The operator CLI ships a stella trust command group with exactly these subcommands: init, sync, status, verify, export, import, and snapshot export (src/Cli/StellaOps.Cli/Commands/Trust/TrustCommandGroup.cs). There is no stella rekor group and no reset-circuits subcommand. Several commands in earlier drafts of this guide named flags that do not exist; those have been corrected inline below.

Overview

This guide covers disaster recovery procedures for Stella Ops trust infrastructure, including Rekor outages, key compromise, and TUF repository failures.

Scenario 1: Rekor Service Outage

Symptoms

• Attestation submissions failing
• Verification requests timing out
• Circuit breaker reporting OPEN state

Immediate Actions

1. Verify the outage

   # Check Rekor health
   curl -sf https://rekor.sigstore.dev/api/v1/log | jq .
   

   The Rekor circuit breaker is a server-side component inside the Attestor (StellaOps.Attestor.Core.Resilience.CircuitBreaker, wrapped by ResilientRekorClient). Its state (Closed / Open / HalfOpen) is not exposed by the stella CLI — there is no stella trust status --show-circuit-breaker flag (the only stella trust status flags are --show-keys/-k, --show-endpoints/-e, and --output/-o). Observe the breaker via the Attestor service logs, which emit Circuit {Name} transitioned from {OldState} to {NewState} and Primary Rekor circuit breaker: {OldState} -> {NewState} on every transition.

2. Check if mirror failover engaged

   The Attestor’s ResilientRekorClient performs automatic mirror failover when the primary circuit opens (it routes operations to the configured mirror with no operator action). There is no stella trust status --show-backends command. Confirm failover from the Attestor logs (Primary circuit is OPEN, routing {Operation} to mirror) and from whether the mirror is configured in AttestorOptions (Rekor mirror settings) before relying on it.

3. If the mirror is not available, swap endpoints via TUF

   # On TUF repository admin system
   ./devops/scripts/disaster-swap-endpoint.sh \
     --repo /path/to/tuf \
     --new-rekor-url https://rekor-mirror.internal:8080 \
     --note "Emergency: Production Rekor outage $(date -u)"
   

   The script prompts for an interactive confirmation: type SWAP to proceed. Accepted flags are --repo, --new-rekor-url (both required), --new-fulcio-url, --note, and --version (auto-increments the service-map version if omitted). It writes a new sigstore-services-v{N}.json target; it does not sign or publish the metadata — that is the next step.

4. Sign and publish the updated metadata

   The swap script only stages the new service map. You must then sign the updated TUF metadata (targets.json → snapshot.json → timestamp.json, each with its respective key) and deploy it to the TUF server. These signing and publish steps use the TUF repository’s own tooling (e.g. the scripts/sign-metadata.sh / scripts/publish.sh that live in your TUF repo directory); they are not commands shipped by StellaOps. The swap script prints the exact required next steps when it completes.

5. Force client sync (optional, for immediate effect)

   stella trust sync --force
   

Key Principle

No client reconfiguration required. Endpoint changes flow through TUF. Clients discover new endpoints within their configured refresh interval.

Recovery

Once the primary Rekor is restored:

1. Swap back to primary

   ./devops/scripts/disaster-swap-endpoint.sh \
     --repo /path/to/tuf \
     --new-rekor-url https://rekor.sigstore.dev \
     --note "Recovery: Primary Rekor restored"
   

2. Verify the service map published

   stella trust sync --force
   stella trust status --show-endpoints
   

3. Circuit breakers recover automatically

   There is no stella trust reset-circuits command. The Attestor circuit breaker self-heals: once a primary is reachable again it transitions Open → HalfOpen automatically after OpenDurationSeconds and back to Closed after SuccessThreshold consecutive successful probes (see CircuitBreaker.cs). A Reset() to force-close exists only as an in-process API on ResilientRekorClient/CircuitBreaker; it is not exposed to operators. To force an immediate re-probe you can restart the Attestor service.

Scenario 2: Rekor Key Compromise

Symptoms

• Security team reports potential key exposure
• Unusual entries in transparency log
• Third-party security advisory

Immediate Actions

1. Assess the compromise scope

   • When was the key potentially exposed?
   • What entries may be affected?
   • Are there signed entries from the compromised period?

2. Emergency key rotation

   # Phase 1: Add new key immediately (no grace period)
   ./devops/scripts/rotate-rekor-key.sh add-key \
     --repo /path/to/tuf \
     --new-key /secure/new-rekor-key-v2.pub
   
   # Sign and publish immediately
   cd /path/to/tuf
   ./scripts/sign-metadata.sh
   ./scripts/publish.sh
   

3. Force all clients to sync

   • Announce emergency update to all teams
   • Clients should run: stella trust sync --force

4. Revoke compromised key immediately

   # Phase 2: Remove old key (skip grace period due to compromise)
   ./devops/scripts/rotate-rekor-key.sh remove-old \
     --repo /path/to/tuf \
     --old-key-name rekor-key-v1
   
   # Sign and publish (use your TUF repo's own signing/publish tooling)
   cd /path/to/tuf
   ./scripts/sign-metadata.sh
   ./scripts/publish.sh
   

   remove-old requires --repo and --old-key-name, is interactive (prompts Type 'CONFIRM' to proceed), and is irreversible. It removes the key file from targets/ but the script itself does not rewrite targets.json or sign/publish — it reminds you to update targets.json and then sign and publish via your TUF repo’s tooling. add-key accepts --repo, --new-key, optional --new-key-name (defaults to rekor-key-v{N+1}), and --grace-days (default 7).

5. Document the incident

   • Log rotation time
   • Affected key ID and fingerprint
   • List of potentially affected entries
   • Remediation steps taken

Forensics

Identify entries signed during the compromise window.

NOT IMPLEMENTED: There is no stella rekor query command in the CLI (the stella CLI ships a stella trust group — init, sync, status, verify, export, import, snapshot export — but no rekor group). Perform time-range forensics directly against the Rekor transparency log API (e.g. the upstream rekor-cli or the Rekor REST API), correlating entries by the compromised key’s public-key fingerprint:

# Example using the upstream Rekor REST API (not a StellaOps command)
curl -sf "https://rekor.sigstore.dev/api/v1/index/retrieve" \
  -H "Content-Type: application/json" \
  -d '{"publicKey": {"format": "x509", "content": "<base64-pubkey>"}}' | jq .

Cross-reference the returned entry indices against the compromise window using each entry’s integratedTime.

Scenario 3: TUF Repository Unavailable

Symptoms

• Clients cannot sync trust metadata
• stella trust sync failing with network errors
• TUF timestamp verification failing

Immediate Actions

1. Diagnose the issue

   # Check TUF repository health
   curl -sf https://trust.example.com/tuf/timestamp.json | jq .
   
   # Check DNS resolution
   nslookup trust.example.com
   
   # Check TLS certificate
   openssl s_client -connect trust.example.com:443 -servername trust.example.com
   

2. For clients - tolerate stale metadata

   stella trust sync has no --allow-stale / --max-age flags (its only options are --force/-f and --output/-o). Staleness tolerance is enforced at import time, not sync time: the --reject-if-stale threshold on stella trust import (e.g. --reject-if-stale 7d) decides whether an offline bundle is accepted. When the TUF server is unreachable, clients continue to verify against their already-cached trust state (under ~/.local/share/StellaOps/TufCache) until a fresh sync succeeds; there is no flag to relax the live freshness check on sync. To re-seed from a known-good offline bundle while the server is down, import with an explicit staleness bound:

   stella trust import ./trust-bundle.tar.zst --reject-if-stale 7d
   

3. Restore TUF server

   • Check hosting infrastructure
   • Restore from backup if needed
   • Verify metadata integrity

4. Deploy mirror (if available)

   # Update DNS or load balancer to point to mirror
   # Or update clients directly (less preferred)
   stella trust init \
     --tuf-url https://trust-mirror.example.com/tuf/ \
     --force
   

Scenario 4: Signing Key Compromise

Symptoms

• Security team reports key exposure
• Unauthorized attestations appearing

Immediate Actions

1. Generate the replacement key first

   ./devops/scripts/rotate-signing-key.sh generate \
     --key-type ecdsa-p256
   

   Valid --key-type values are ecdsa-p256 (default), ecdsa-p384, and rsa-4096. Keys are written under --key-dir (default /etc/stellaops/keys) as signing-key-v{N}.pem / .pub. Generate the new key before retiring the old one so signing capability is never lost.

2. Retire the compromised key

   ./devops/scripts/rotate-signing-key.sh retire \
     --old-key compromised-key-name
   

   retire requires --old-key and is interactive and irreversible: it prompts Type 'RETIRE' to proceed and then archives (does not delete) the old key under <key-dir>/archived/. The script’s normal flow is generate → activate → verify → retire with a default 14-day grace period; in a confirmed-compromise scenario you skip the grace period and retire immediately, but you still complete the CI/CD and notification steps below.

3. Update CI/CD immediately

   • Remove compromised key from all pipelines
   • Add new key
   • Trigger rebuild of recent releases

4. Notify downstream consumers

   • Announce key rotation
   • Provide new public key
   • Advise re-verification of recent attestations

Scenario 5: Root Key Ceremony Required

When Required

• Scheduled root key rotation (typically annual)
• Root key compromise (emergency)
• Threshold change for root signatures

Procedure

1. Schedule ceremony

   • Require M-of-N key holders present
   • Air-gapped ceremony machine
   • Hardware security modules

2. Generate new root

   NOT IMPLEMENTED (no shipped tool): StellaOps does not ship a tuf-ceremony binary — there is no such command anywhere in src/ or devops/. Generate the new root with your standard TUF tooling (tuf/python-tuf, cosign/sigstore root signing, or your HSM-backed ceremony procedure). The illustrative command below is not a StellaOps command:

   # Illustrative only — use your organization's TUF ceremony tooling
   tuf-ceremony init --threshold 3 --keys 5 --algorithm ed25519
   

3. Sign new root with old keys

   • Requires old threshold of signatures
   • Ensures continuous trust chain

4. Distribute new root

   • Publish to TUF repository
   • Update bootstrap documentation
   • Notify all operators

Air-Gap Considerations

For air-gapped deployments after root rotation:

# Export a trust bundle for offline distribution.
# NOTE: there is no `--include-root` flag. `stella trust snapshot export`
# accepts: --out/-o (required), --from-proxy, --tiles, --include-entries, --depth.
# The exported snapshot carries the current TUF metadata (including the root that
# is part of the synced trust state) — it does not need an explicit root flag.
stella trust snapshot export \
  --out post-rotation-bundle.tar.zst

# Alternatively, export the plain trust state (TUF metadata + targets):
#   stella trust export --out ./post-rotation-bundle/

# Transfer and import on air-gapped systems
./devops/scripts/bootstrap-trust-offline.sh \
  post-rotation-bundle.tar.zst \
  --force  # Required due to root change

The offline bootstrap script imports the bundle via stella trust import (--verify-manifest is on by default) and then verifies with stella trust status --show-keys. --force is passed through to the import so a root change does not fail manifest validation.

Communication Templates

Outage Notification

Subject: [StellaOps] Rekor Service Disruption - Failover Active

Status: Service Degradation
Impact: Attestation submissions may be delayed
Mitigation: Automatic failover to mirror active

Action Required: None - clients will auto-discover new endpoint

Updates: Monitor status at https://status.example.com

Key Rotation Notice

Subject: [StellaOps] Emergency Key Rotation - Action Required

Reason: Security precaution / Scheduled rotation
Affected Key: rekor-key-v1 (fingerprint: abc123...)
New Key: rekor-key-v2 (fingerprint: def456...)

Action Required:
1. Run: stella trust sync --force
2. Verify: stella trust status --show-keys

Timeline: Old key will be revoked at [DATE/TIME UTC]

Monitoring and Alerting

Key Metrics

• Circuit breaker state changes
• TUF metadata freshness
• Rekor submission latency
• Verification success rate

Alert Thresholds

Contacts

Related Documentation

• Bootstrap Guide
• Key Rotation Runbook
• TUF Integration Guide