Air-Gap Operations Runbook

Version: 1.1.2
Sprint: 3500.0004.0004
Last Updated: 2026-05-31

Audience: operators running Stella Ops in air-gapped (offline) environments. This runbook covers operational procedures for offline kit management, feed updates, scanning, and isolated verification workflows.

Read this banner first. Much of the CLI and configuration shown below predates the shipped code. The reconciliation note immediately following enumerates what is actually wired today; each section is annotated inline as implemented, Draft/roadmap, or NOT IMPLEMENTED. Use the CLI Quick Reference as the source of truth for runnable commands.

Implementation status (reconciled against src/ 2026-05-31). Large parts of this runbook describe an aspirational stella offline-kit … CLI and a generic mode: air-gap configuration model that do not exist in the codebase. The commands and config that are actually wired today are narrower:

• CLI: the live command groups are stella airgap (subcommands import, seal, export-evidence) and stella offline (subcommands import, status). There is no stella offline-kit, stella feeds age, stella config set mode, stella health check --air-gap, stella score replay, or stella proof verify command on the air-gap path. A richer airgap group (export/import/diff/status/jobs) exists in src/Cli/StellaOps.Cli/Commands/AirGapCommandGroup.cs but is not registered in the CLI root (CommandFactory), so it is not reachable from stella today.
• HTTP API (AirGap Controller, src/AirGap/StellaOps.AirGap.Controller): GET /system/airgap/status (scope airgap:status:read), POST /system/airgap/seal and POST /system/airgap/unseal (scope airgap:seal), POST /system/airgap/verify (scope airgap:status:read). Tenant is taken from the JWT stellaops:tenant claim (with an optional matching x-tenant-id header). Scope note: the verify endpoint’s AirGap.Verify policy requires StellaOpsScopes.AirgapStatusRead (Program.cs:59-69) — verification is a read-only integrity check, so it shares the air-gap read scope rather than a distinct one. There is no airgap:verify scope; the canonical catalog (StellaOpsScopes.cs) defines exactly airgap:seal, airgap:import, and airgap:status:read. Since airgap:status:read is already granted to the console and CLI clients, POST /system/airgap/verify is reachable with an ordinary OpTok. Note also that airgap:import is registered as a scope but the controller’s AirGap.Import policy is not bound to any controller route.
• HTTP API (AirGap Time, src/AirGap/StellaOps.AirGap.Time): GET /api/v1/time/status (scope airgap:status:read), POST /api/v1/time/anchor (scope airgap:seal — there is no dedicated anchor-write scope; AirGapTimePolicies.AnchorWriteScope reuses airgap:seal), readiness at /healthz/ready. Tenant is taken from the JWT stellaops:tenant claim; a tenantId query/body value, if present, must match the claim.
• Mirror import (Concelier): the real endpoint is POST /api/v1/advisory-sources/mirror/import (status at …/mirror/import/status), not POST /api/v1/mirror/import.
• Egress model: sealing is an egress policy with two modes — Unsealed (advisory) and Sealed (block by allow-rule) — see src/AirGap/StellaOps.AirGap.Policy. There is no disableNetworking flag.

Inline annotations below mark each block as implemented, Draft/roadmap, or NOT IMPLEMENTED so operators do not run commands that will fail.

Table of Contents

1. Overview
2. Offline Kit Management
3. Feed Updates
4. Scanning in Air-Gap Mode
5. Verification in Air-Gap Mode
6. Troubleshooting
7. Monitoring & Health Checks
8. Escalation Procedures

1. Overview

What is Air-Gap Mode?

Air-gap mode allows StellaOps to operate in environments with no external network connectivity. This is required for:

• Classified or sensitive environments
• High-security facilities
• Regulatory compliance (certain industries)
• Disaster recovery scenarios

Air-Gap Architecture

┌─────────────────────────────────────────────────────────────┐
│                 Connected Environment                       │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐     │
│  │ Feed Sync   │───►│ Bundle      │───►│ Offline Kit │     │
│  │ Service     │    │ Generator   │    │ (.tar.gz)   │     │
│  └─────────────┘    └─────────────┘    └──────┬──────┘     │
└─────────────────────────────────────────────────┼───────────┘
                                                  │ Physical
                                                  │ Transfer
┌─────────────────────────────────────────────────┼───────────┐
│                 Air-Gapped Environment          │           │
│  ┌──────────────┐    ┌─────────────┐    ┌──────▼──────┐    │
│  │ StellaOps    │◄───│ Offline     │◄───│ Import      │    │
│  │ Scanner      │    │ Data Store  │    │ Service     │    │
│  └──────────────┘    └─────────────┘    └─────────────┘    │
└─────────────────────────────────────────────────────────────┘

Offline Kit Contents

2. Offline Kit Management

NOT IMPLEMENTED — illustrative CLI. The stella offline-kit … command family (create, import, status, list, verify, rollback, snapshots, days-until-expiry) does not exist in src/Cli. Offline-kit import is done today with stella offline import --bundle <path.tar.zst> (see src/Cli/StellaOps.Cli/Commands/OfflineCommandGroup.cs); air-gap mirror-bundle import is stella airgap import --bundle <dir> (CommandFactory.BuildAirgapCommand). The snippets in 2.1–2.4 are kept as a conceptual workflow but must be mapped to the live commands in the CLI Quick Reference before use.

The Console’s read-only status surface is implemented at /ops/operations/offline-kit. Its Dashboard reads the tenant-scoped Scanner /api/v1/offline-kit/status and /api/v1/offline-kit/manifest contracts and shows a bounded loading, honest empty, or explicit error state; it does not synthesize bundle or activity data in the browser.

2.1 Generating an Offline Kit

On the connected system:

# Generate full offline kit
stella offline-kit create \
  --output /path/to/offline-kit.tar.gz \
  --include-all

# Generate minimal kit (feeds only)
stella offline-kit create \
  --output /path/to/offline-kit-feeds.tar.gz \
  --feeds-only

# Generate with specific components
stella offline-kit create \
  --output /path/to/offline-kit.tar.gz \
  --include vuln-db \
  --include advisories \
  --include trust-bundles \
  --include rules

2.2 Kit Manifest

Each kit includes a manifest for verification:

{
  "version": "1.0.0",
  "createdAt": "2025-01-15T00:00:00Z",
  "expiresAt": "2025-02-15T00:00:00Z",
  "components": {
    "vulnerability-database": {
      "hash": "sha256:abc123...",
      "size": 1073741824,
      "records": 245000
    },
    "advisory-feeds": {
      "hash": "sha256:def456...",
      "size": 536870912,
      "lastUpdate": "2025-01-15T00:00:00Z"
    },
    "trust-bundles": {
      "hash": "sha256:ghi789...",
      "size": 65536,
      "certificates": 12
    },
    "signing-keys": {
      "hash": "sha256:jkl012...",
      "keyIds": ["key-001", "key-002"]
    }
  },
  "signature": "base64-signature..."
}

2.3 Transferring to Air-Gapped Environment

Physical Media Transfer

# On connected system - write to media
cp offline-kit.tar.gz /media/secure-usb/
sha256sum offline-kit.tar.gz > /media/secure-usb/offline-kit.tar.gz.sha256

# On air-gapped system - verify and import
cd /media/secure-usb/
sha256sum -c offline-kit.tar.gz.sha256
stella offline-kit import --kit offline-kit.tar.gz

Secure File Transfer (if available)

# Using data diode or one-way transfer
scp offline-kit.tar.gz airgap-gateway:/incoming/

2.4 Installing Offline Kit

# Import and install kit
stella offline-kit import \
  --kit /path/to/offline-kit.tar.gz \
  --verify \
  --install

# Verify installation
stella offline-kit status

# List installed components
stella offline-kit list

Mirror import (Concelier). Concelier exposes a durable mirror import endpoint backed by a filesystem import path: POST /api/v1/advisory-sources/mirror/import (import status at GET /api/v1/advisory-sources/mirror/import/status). This is the canonical endpoint — the previously documented POST /api/v1/mirror/import path and “mirror.md section 8.4” reference are both incorrect (the current docs/modules/concelier/operations/mirror.md has no section 8.4). The wizard step name and a UI-driven import flow are Draft/roadmap; verify against the Console before relying on them.

Expected output:

Offline Kit Status
══════════════════════════════════════════
Mode:           AIR-GAP
Kit Version:    1.0.0
Installed At:   2025-01-15T10:30:00Z
Expires At:     2025-02-15T00:00:00Z

Components:
  ✓ vulnerability-database  2025-01-15  245,000 records
  ✓ advisory-feeds          2025-01-15  Active
  ✓ trust-bundles           2025-01-15  12 certificates
  ✓ signing-keys            2025-01-15  2 keys
  ✓ rules-engine            2025-01-15  v2.3.0

Health: HEALTHY
Days Until Expiry: 31

3. Feed Updates

NOT IMPLEMENTED — illustrative CLI. stella offline-kit create --delta-from/--delta-to, stella offline-kit snapshots, and stella offline-kit rollback have no backing command. stella feeds status resolves to the deterministic-replay feed-snapshot group (src/Cli/StellaOps.Cli/Commands/FeedsCommandGroup.cs) / admin feeds group, not an air-gap freshness reporter. Delta kits and snapshot rollback are Draft/roadmap; the supported recovery action today is re-importing the last known-good bundle (see docs/modules/airgap/guides/operations.md “Failure recovery”).

3.1 Update Workflow

┌────────────────────────────────────────────────────────────┐
│ Weekly Feed Update Process                                 │
├────────────────────────────────────────────────────────────┤
│ Day 1: Generate kit on connected system                    │
│ Day 2: Security review and approval                        │
│ Day 3: Transfer to air-gapped environment                  │
│ Day 4: Import and verify                                   │
│ Day 5: Activate new feeds                                  │
└────────────────────────────────────────────────────────────┘

3.2 Generating Delta Updates

For faster updates, generate delta kits:

# On connected system
stella offline-kit create \
  --output delta-kit.tar.gz \
  --delta-from 2025-01-08 \
  --delta-to 2025-01-15

# Delta kit is smaller, contains only changes

3.3 Applying Updates

# Import delta update
stella offline-kit import \
  --kit delta-kit.tar.gz \
  --delta \
  --verify

# Verify feed freshness
stella feeds status

3.4 Rollback Procedure

If an update causes issues:

# List available snapshots
stella offline-kit snapshots

# Rollback to previous version
stella offline-kit rollback --to 2025-01-08

# Verify rollback
stella feeds status

4. Scanning in Air-Gap Mode

NOT IMPLEMENTED as written — see notes. There is no global stella config set mode air-gap toggle and no stella config get mode that returns air-gap. Sealing is enforced per-process through the egress policy (EgressPolicyMode = Unsealed/Sealed, src/AirGap/StellaOps.AirGap.Policy) and, for the controller, via POST /system/airgap/seal. Scan invocation flags shown below (stella scan image --local, --offline-feeds) are Draft/roadmap for the air-gap surface; the live scan command group is src/Cli/StellaOps.Cli/Commands/CommandFactory.cs (scan) — verify its actual flags before use.

4.1 Enabling Air-Gap Mode

# Enable air-gap mode
stella config set mode air-gap

# Verify mode
stella config get mode
# Output: air-gap

4.2 Running Scans

# Scan a local image (no registry pull)
stella scan image --local /path/to/image.tar

# Scan from local registry
stella scan image localhost:5000/myapp:v1.0

# Scan with offline feeds explicitly
stella scan image myapp:v1.0 --offline-feeds

4.3 Local Image Preparation

For images that need to be scanned:

# On connected system - save image
docker save myapp:v1.0 -o myapp-v1.0.tar
sha256sum myapp-v1.0.tar > myapp-v1.0.tar.sha256

# Transfer to air-gapped system
# ... physical transfer ...

# On air-gapped system - verify and load
sha256sum -c myapp-v1.0.tar.sha256
docker load -i myapp-v1.0.tar
stella scan image myapp:v1.0 --local

4.4 SBOM Generation

# Generate SBOM for local image
stella sbom generate --image myapp:v1.0 --output sbom.json

# Scan existing SBOM
stella scan sbom --file sbom.json

5. Verification in Air-Gap Mode

Mixed implementation. Offline verification commands exist but under different names/groups than shown here, and the closest matches differ in how “real” they are:

• stella verify offline(VerifyCommandGroup, registered) — live with a real handler. This is the actual offline-evidence verification command.
• stella score verify(the score command is built by ScoreReplayCommandGroup.BuildScoreCommand, registered — not the unregistered ScoreCommandGroup) — live; calls a real score-bundle verify endpoint.
• stella evidence proof / stella evidence replay score(EvidenceCommandGroup, registered) — present but the subcommands are stub placeholders that print canned Console.WriteLine output and return 0 without verifying anything. There is no stella evidence proof verify subcommand (the proof group only has generate/anchor/receipt). Do not rely on these for real verification yet.
• stella trust …(Commands/Trust/TrustCommandGroup.cs) — NOT reachable: BuildTrustCommand is defined but is not registered in the CLI root (CommandFactory); it is only wired up inside a unit test. So stella trust verify, stella trust export, and stella trust import are all unreachable today. (For the record, the defined-but-unwired group does contain init/sync/status/verify/export/import/snapshot export TUF operations — so export/import exist in code, they just are not exposed.)

The specific spellings below — stella proof verify, stella score replay --offline, stella trust export, stella trust import — do not all map to live commands. Treat 5.1–5.3 as Draft/roadmap spellings and confirm against stella <group> --help.

5.1 Offline Proof Verification

# Verify proof bundle offline
stella proof verify --bundle bundle.tar.gz --offline

# Verify with explicit trust store
stella proof verify --bundle bundle.tar.gz \
  --offline \
  --trust-store /etc/stellaops/offline/trust-roots.json

5.2 Preparing Trust Store

Before air-gapped deployment:

# On connected system - export trust configuration
stella trust export \
  --output trust-roots.json \
  --include-ca \
  --include-signing-keys

# Transfer to air-gapped system
# ... physical transfer ...

# On air-gapped system - import trust
stella trust import --file trust-roots.json

5.3 Score Replay Offline

# Replay score using offline data
stella score replay --scan $SCAN_ID --offline

# Replay with frozen time
stella score replay --scan $SCAN_ID \
  --offline \
  --freeze 2025-01-15T00:00:00Z

6. Troubleshooting

CLI spellings are illustrative. The diagnostic commands in this section (stella offline-kit verify, stella feeds status, stella trust list, stella trust check-expiry, stella config get mode) follow the aspirational CLI and are NOT IMPLEMENTED as written. The diagnostic intent is sound; map to live commands: mirror-bundle verification → stella airgap import --verify-only (the airgap import group is the only one with a --verify-only flag); offline-kit validation → stella offline import --bundle <path> --dry-run (the offline import group has no --verify-only; it validates without activating via --dry-run, with DSSE/Rekor checks on by default); air-gap state → GET /system/airgap/status; time-anchor staleness → GET /api/v1/time/status. The openssl verify … and journalctl/grep steps are generic OS tooling and remain valid.

6.1 Kit Import Fails

Symptoms: Failed to import offline kit

Diagnostic Steps:

1. Verify kit integrity:

   sha256sum offline-kit.tar.gz
   # Compare with manifest
   

2. Check kit signature:

   stella offline-kit verify --kit offline-kit.tar.gz
   

3. Check disk space:

   df -h /var/lib/stellaops/
   

Common Causes:

6.2 Stale Feed Data

Symptoms: Scans report old vulnerabilities, miss new CVEs

Diagnostic Steps:

1. Check feed age:

   stella feeds status
   

2. Verify last update:

   stella offline-kit status | grep "Installed At"
   

Resolution:

• Generate and import fresh offline kit
• Establish regular update schedule
• Set up expiry alerts

6.3 Trust Verification Fails

Symptoms: Certificate chain verification failed in offline mode

Diagnostic Steps:

1. Check trust store:

   stella trust list
   

2. Verify CA bundle:

   openssl verify -CAfile /etc/stellaops/offline/ca-bundle.pem \
     /path/to/certificate.pem
   

3. Check for expired roots:

   stella trust check-expiry
   

Resolution:

• Update trust bundle in offline kit
• Import new CA certificates
• Rotate expired signing keys

6.4 Network Access Attempted

Symptoms: Air-gapped system attempts network connection

Diagnostic Steps:

1. Check mode configuration:

   stella config get mode
   

2. Audit network attempts:

   journalctl -u stellaops | grep -i "network\|connect\|http"
   

3. Verify no external URLs in config:

   grep -r "http" /etc/stellaops/
   

Resolution:

• Ensure mode: air-gap in configuration
• Remove any hardcoded URLs
• Block outbound traffic at firewall level

7. Monitoring & Health Checks

stella health check --air-gap is NOT IMPLEMENTED. No CLI health subcommand emits the consolidated air-gap report shown in 7.1, and the automated script in 7.2 calls non-existent commands (stella feeds age --days, stella offline-kit days-until-expiry, stella trust verify). Service readiness is exposed at /healthz/ready (AirGap Time) and air-gap state at GET /system/airgap/status. The actual emitted metrics are enumerated in 7.3 (corrected); the example script and alert YAML in 7.2/7.4 are Draft/roadmap illustrations.

7.1 Air-Gap Health Checks

# Run comprehensive health check
stella health check --air-gap

# Output
Air-Gap Health Check
══════════════════════════════════════════
✓ Mode:                 air-gap
✓ Feed Freshness:       7 days old (OK)
✓ Trust Store:          Valid
✓ Signing Keys:         2 active
✓ Disk Space:           45% used
✓ Database:             Healthy
⚠ Kit Expiry:           24 days remaining

Overall: HEALTHY (1 warning)

7.2 Automated Monitoring Script

#!/bin/bash
# /etc/stellaops/scripts/airgap-health.sh

set -e

# Check feed age
FEED_AGE=$(stella feeds age --days)
if [ "$FEED_AGE" -gt 14 ]; then
  echo "CRITICAL: Feeds are $FEED_AGE days old"
  exit 2
fi

# Check kit expiry
DAYS_LEFT=$(stella offline-kit days-until-expiry)
if [ "$DAYS_LEFT" -lt 7 ]; then
  echo "WARNING: Kit expires in $DAYS_LEFT days"
  exit 1
fi

# Check trust store
if ! stella trust verify --quiet; then
  echo "CRITICAL: Trust store verification failed"
  exit 2
fi

echo "OK: Air-gap health check passed"
exit 0

7.3 Metrics for Air-Gap

Corrected to match emitted instruments. The previously listed metric names (offline_kit_age_days, offline_kit_expiry_days, feed_freshness_days, trust_store_valid, disk_usage_percent) are NOT emitted by any service in src/. The real instruments are below, sourced from the three air-gap meters. Note the units: ages/budgets are reported in seconds, not days, so alert thresholds must be expressed in seconds.

Meter StellaOps.AirGap.Controller(AirGapTelemetry.cs):

Meter StellaOps.AirGap.Importer(OfflineKitMetrics.cs):

Meter StellaOps.AirGap.Sync(HLC job sync — AirGapSyncMetrics.Counters.cs). The full instrument set is:

The condensed table in Appendix D Monitoring Metrics lists only the five most operationally relevant of these.

7.4 Alert Configuration

# /etc/stellaops/alerts/airgap.yaml
alerts:
  - name: offline_kit_expiring
    condition: offline_kit_expiry_days < 7
    severity: warning
    message: "Offline kit expires in {{ .Value }} days"

  - name: offline_kit_expired
    condition: offline_kit_expiry_days <= 0
    severity: critical
    message: "Offline kit has expired"

  - name: feeds_stale
    condition: feed_freshness_days > 14
    severity: warning
    message: "Vulnerability feeds are {{ .Value }} days old"

8. Escalation Procedures

8.1 Escalation Matrix

8.2 Emergency Kit Update Process

When kit expires before scheduled update:

1. On Connected System (0-2 hours):

   stella offline-kit create --output emergency-kit.tar.gz --include-all
   

2. Security Review (2-4 hours):

   • Verify kit signature
   • Check for known vulnerabilities in kit
   • Get approval for transfer

3. Transfer (4-6 hours):

   • Physical media preparation
   • Chain of custody documentation
   • Transfer to air-gapped environment

4. Import (6-8 hours):

   stella offline-kit import --kit emergency-kit.tar.gz --verify --install
   

8.3 Contacts

Appendix A: Configuration Reference

Draft/roadmap — does not match live config keys. No service binds a top-level mode: air-gap, an offline: block (dataDir, feedsDir, trustStore, caBundle, disableNetworking, kit:, feeds:), or the STELLAOPS_MODE / STELLAOPS_DISABLE_NETWORKING environment variables. The real sealing config is the egress policy section consumed by src/AirGap/StellaOps.AirGap.Policy:

egressPolicy:            # bound via AddEgressPolicy(...); section name is host-defined
  mode: Sealed           # Unsealed (advisory) | Sealed (block unless allow-ruled)
  allowLoopback: true
  allowPrivateNetworks: false
  allowRules:
    - hostPattern: registry.internal
      port: 5000
      transport: Any
  remediationDocumentationUrl: https://…
  supportContact: airgap-ops@…

AirGap Time options (AirGapOptions, src/AirGap/StellaOps.AirGap.Time/Models): TenantId, Staleness.{WarningSeconds,BreachSeconds}, per-content ContentBudgets (advisories/vex/policy), TrustRootFile, AllowUntrustedAnchors, AllowRequestSuppliedTrustRoots. The YAML below is retained only as an aspirational example.

Air-Gap Mode Configuration

# /etc/stellaops/config.yaml
mode: air-gap

offline:
  dataDir: /var/lib/stellaops/offline
  feedsDir: /var/lib/stellaops/offline/feeds
  trustStore: /etc/stellaops/offline/trust-roots.json
  caBundle: /etc/stellaops/offline/ca-bundle.pem
  
  # Disable all external network calls
  disableNetworking: true
  
  # Kit expiry settings
  kit:
    expiryWarningDays: 7
    maxAgeDays: 30
    
  # Feed freshness settings  
  feeds:
    maxAgeDays: 14
    warnAgeDays: 7

Environment Variables

# Air-gap specific
export STELLAOPS_MODE=air-gap
export STELLAOPS_OFFLINE_DATA_DIR=/var/lib/stellaops/offline
export STELLAOPS_DISABLE_NETWORKING=true

# Trust configuration
export STELLAOPS_TRUST_STORE=/etc/stellaops/offline/trust-roots.json
export STELLAOPS_CA_BUNDLE=/etc/stellaops/offline/ca-bundle.pem

Appendix B: CLI Quick Reference

Reconciled against src/Cli/StellaOps.Cli (registered commands only). The list below reflects what is actually reachable from the stella root command. Commands from earlier revisions of this runbook (stella offline-kit …, stella feeds age, stella health check --air-gap, stella config get mode, stella proof verify --offline, stella score replay --offline) are NOT IMPLEMENTED and were removed. The whole stella trust … group is defined in code but not registered in the CLI root, so it is unreachable today (see the “NOT reachable” note below).

# Air-gap environment ops (CommandFactory.BuildAirgapCommand)
stella airgap import --bundle <dir>           # Import a mirror bundle into the local store
stella airgap import --bundle <dir> --verify-only   # Verify integrity without importing
stella airgap seal [--reason <text>]          # Seal the environment for air-gapped operation
stella airgap export-evidence -o <dir>        # Export portable evidence packages

# Offline kit import (OfflineCommandGroup)
stella offline import --bundle <path.tar.zst> # Import offline kit with DSSE/Rekor verification
stella offline import --bundle <path> --dry-run     # Validate without activating
stella offline status                         # Show current offline kit status

# Verification (different groups; confirm with --help)
stella verify offline …                       # Offline evidence verification (VerifyCommandGroup) — REAL handler
stella score verify …                         # Verify a score bundle (score cmd built by ScoreReplayCommandGroup) — REAL handler
stella evidence proof generate|anchor|receipt # Evidence proof ops (EvidenceCommandGroup) — STUB placeholders, no real verification; NO `proof verify` subcommand
stella evidence replay score …                # Score-replay verification (EvidenceCommandGroup) — STUB placeholder

# AirGap Controller HTTP API (no CLI wrapper today)
#   GET  /system/airgap/status
#   POST /system/airgap/seal      | /unseal      | /verify
# AirGap Time HTTP API
#   GET  /api/v1/time/status      | POST /api/v1/time/anchor | GET /healthz/ready

# NOT reachable from the `stella` root today:
#   stella trust init|sync|status|verify|export|import|snapshot
#     (TrustCommandGroup.BuildTrustCommand is defined but NOT registered in CommandFactory)
# NOT registered in the CLI root (exists in AirGapCommandGroup.cs but unwired):
#   stella airgap export | diff | status | jobs export|import|list

Appendix C: Update Schedule Template

Revision History

Appendix D: HLC-Based Job Synchronization

Sprint: SPRINT_20260105_002_003_ROUTER
Added: 2026-01-07

Status: implemented core, CLI not wired. The HLC sync engine and its telemetry are real — see src/AirGap/__Libraries/StellaOps.AirGap.Sync and the command handlers in src/Cli/StellaOps.Cli/Commands/CommandHandlers.AirGap.cs. The Monitoring Metrics names below match the emitted StellaOps.AirGap.Sync meter exactly. However, the stella airgap export and stella airgap jobs export|import|list commands live in AirGapCommandGroup.cs, which is not registered in CommandFactory, so they are not reachable from the stella root today (only airgap import, airgap seal, airgap export-evidence are). The --conflict-strategy and stella airgap verify --bundle references in the troubleshooting subsection are likewise Draft/roadmap until the group is wired in.

Overview

When running in air-gap mode, scheduled jobs are queued locally using Hybrid Logical Clocks (HLC). Upon reconnection or physical transfer, these jobs can be merged deterministically with the central scheduler while preserving global ordering.

Key Concepts

Exporting Jobs from Offline Node

# Export all pending jobs to a bundle
stella airgap export \
  --output /media/usb/jobs-bundle.json \
  --tenant <tenant-id>

# Export with DSSE signature (requires signing key)
stella airgap export \
  --output /media/usb/jobs-bundle.json \
  --tenant <tenant-id> \
  --sign

Importing Jobs to Central Scheduler

# Import bundle and merge jobs
stella airgap import \
  --bundle /media/usb/jobs-bundle.json

# Import with signature verification
stella airgap import \
  --bundle /media/usb/jobs-bundle.json \
  --verify-signature

# Dry-run to preview merge results
stella airgap import \
  --bundle /media/usb/jobs-bundle.json \
  --dry-run

Merge Behavior

1. Total Ordering: Jobs are merged by HLC order key (PhysicalTime, LogicalCounter, NodeId, JobId)
2. Duplicate Detection: Jobs with the same JobId are deduplicated
3. Chain Verification: Each job’s chain link is verified against its predecessor
4. Conflict Resolution: Same-payload jobs from different nodes are kept; conflicting payloads are flagged

Monitoring Metrics

These are the most operationally relevant instruments from the StellaOps.AirGap.Sync meter (AirGapSyncMetrics.Counters.cs). The full set — which also includes airgap_duplicates_dropped_total, airgap_offline_enqueues_total, airgap_bundle_size_bytes, and airgap_merge_entries_count — is enumerated in §7.3 Metrics for Air-Gap.

Troubleshooting

Bundle signature verification fails

1. Check that signing key is correctly configured
2. Verify bundle was not modified during transfer
3. Check key ID matches expected signer

Merge conflicts detected

1. Review conflicts in import output
2. Check for clock skew between offline nodes
3. Use --conflict-strategy flag to auto-resolve

Jobs missing after import

1. Check for duplicate JobIds (same job, same node)
2. Verify chain integrity with stella airgap verify --bundle <path>
3. Review import logs for skipped entries