Export Center Operations Runbook

Audience: DevOps and on-call operators running the StellaOps Export Center service in production and air-gapped environments. Purpose: Provide the operational procedures — monitoring, capacity, failure response, recovery, and verification — for keeping export runs healthy and bundles verifiable.

Some workers and API surfaces are still landing across the Export Center delivery milestones. Validate each command against your deployed version before relying on it; this runbook captures the target procedures.

1. Service scope

The Export Center packages StellaOps evidence and policy overlays into reproducible bundles (JSON, Trivy DB, mirror). Operations owns:

• Worker scaling, queue management, and distribution storage.
• Monitoring and alerts for run throughput, failures, and verification issues.
• Runbook execution for recovery, retention, and compliance.
• Coordination with DevOps validation (cosign + trivy module db import smoke tests).

Related documentation:

• Overview
• Architecture
• Profiles
• Trivy adapter
• Mirror bundles
• API reference
• CLI commands
• KMS envelope pattern
• Risk bundle provider matrix

2. Contacts & tooling

Primary tooling:

• stella export CLI (submit, watch, download, verify).
• Export Center API (/api/export/*) for automation.
• Grafana dashboards (Export Center / Run Health, Export Center / Distribution).
• Alertmanager routes (Export.Center.Failures, Export.Center.Verify).

3. Monitoring & SLOs

Key metrics (exposed by workers and API):

Dashboards must include:

• Run throughput by profile.
• Failure breakdown (adapter, signing, distribution).
• Queue depth and worker concurrency (via Orchestrator metrics).
• Storage consumption (object storage buckets, local staging).

Alerts (Alertmanager):

• ExportCenterRunFailureSpike - exporter_run_failures_total increase rate > 3/15 min.
• ExportCenterVerifyFailure - any entry in exporter_verify_failures_total > 0.
• ExportCenterWorkerLag - queue backlog > threshold for 10 minutes.
• ExportCenterRetentionStale - no pruning events in 24 hours.

4. Routine operations

4.1 Daily checklist

• Review dashboard for run throughput and error classes.
• Confirm CI smoke job (cosign + trivy module db import) passed.
• Check storage usage against capacity thresholds.
• Verify retention job executed (look for exporter_retention_pruned_total increment).
• Scan logs for adapter.trivy.unsupported_schema_version or mirror.delta.apply_failed.

4.2 Weekly tasks

• Rotate Download/OCI API tokens if configured with short-lived credentials.
• Review upcoming profile changes (new tenants, profile updates).
• Test stella export verify against a recent run for each profile.
• Exercise failover of workers (scale to zero one replica, ensure others pick up).

4.3 Pre-release

• Ensure bundles generated for release candidates pass cosign verification.
• Capture sample manifests (export.json, manifest.yaml) for documentation archives.
• Validate Offline Kit packaging includes latest full + delta mirror bundles.

5. Capacity & scaling

5.1 Worker sizing

• Default workers handle ~2 full runs or 6 delta runs concurrently per 4 vCPU.
• Scale out when:
  • Queue depth (exporter_jobs_ready) > 10 for 10 minutes.
  • p95 durations exceed SLO for multiple runs without failures.
• Use Orchestrator quotas: ensure per-tenant concurrency (max_active_runs) is tuned.

5.2 Storage planning

• Staging storage (object store or filesystem) must hold at least:
  • Latest full bundle per tenant per profile.
  • Last N deltas (default N=5).
• Set retention policy via configuration:

ExportCenter:
  Retention:
    Mirror:
      Mode: days
      Value: 30
    Trivy:
      Mode: count
      Value: 10

• Monitor exporter_storage_bytes_total (if available) or use bucket metrics from storage provider.

6. Failure response

Log locations: exporter service emits structured logs with runId, profile, errorCode. For supported deployments, collect logs from Docker Compose or the host service manager.

7. Recovery playbooks

7.1 Replaying a failed run

1. Identify run (runId) and root cause via GET /api/export/runs/{id}.
2. If configuration changed, clone profile and adjust settings.
3. Resubmit run (stella export run submit or API) with --allow-empty if intentionally empty.
4. Monitor SSE stream or stella export run watch.
5. After success, prune failed run data if necessary.

7.2 Restoring from previous full bundle

1. Locate last successful full bundle (mirror:full) and associated manifest.
2. Download and verify signatures.
3. Extract into mirror staging area.
4. Apply subsequent delta bundles in order.
5. Trigger mirror verification script (mirror verify <path>).

7.3 KMS outage response

1. Disable new export submissions temporarily (set per-tenant quota to 0).
2. Coordinate with Authority Core to restore KMS.
3. Once KMS back, run stella export run submit --profile <id> --selectors ... --priority catch-up for affected tenants.

8. Verification workflow

All bundles must pass both signature and content verification.

8.1 Trivy bundle validation (CI job)

cosign verify-blob \
  --key tenants/acme/export-center.pub \
  --signature signatures/trivy-db.sig \
  trivy/db.bundle

trivy module db import trivy/db.bundle --cache-dir /tmp/trivy-cache

Automation: DEVOPS-EXPORT-36-001 ensures this runs on every pipeline.

8.2 Mirror bundle validation

cosign verify-blob \
  --key tenants/acme/export-center.pub \
  --signature signatures/export.sig \
  mirror/export.json

./offline-kit/bin/mirror verify mirror-20251029-full.tar.zst

If encryption enabled, decrypt using age or AES key before verification.

9. Change management

• Profile changes require change record referencing tenant impact and expected bundle size.
• Distribution configuration updates (OCI vs HTTP) must be tested in staging.
• Schema upgrades (e.g., Trivy schema v3) need coordination with DevOps, Exporter, and Docs.
• Update runbook and related docs when processes change (tie updates to DOCS-EXPORT-37-005).

10. References

• Trivy adapter
• Mirror bundles
• KMS envelope pattern
• Risk bundle provider matrix
• Observability
• Aggregation-Only Contract
• Offline Kit