HLC Job Sync Offline Operations

Audience: operators and integrators synchronizing scheduled jobs across disconnected or air-gapped Stella Ops nodes.

This guide describes the offline job-synchronization mechanism, which uses Hybrid Logical Clock (HLC) ordering to merge job logs deterministically across an air gap. For related transfer formats, see Mirror Bundles; for time and staleness rules, see Staleness and Time.

Overview

When nodes operate in disconnected or offline mode, scheduled jobs are enqueued locally with HLC timestamps. Upon reconnection or air-gap transfer, these job logs are merged deterministically to maintain global ordering.

Key features:

• Deterministic ordering: Jobs merge by HLC total order (T_hlc.PhysicalTime, T_hlc.LogicalCounter, NodeId, JobId)
• Chain integrity: Each entry links to the previous via link = Hash(prev_link || job_id || t_hlc || payload_hash)
• Conflict-free: Same payload = same JobId (deterministic), so duplicates are safely dropped
• Audit trail: Source node ID and original links preserved for traceability

CLI Commands

Export Job Logs

Export offline job logs to a sync bundle for air-gap transfer:

# Export job logs for a tenant
stella airgap jobs export --tenant my-tenant -o job-sync-bundle.json

# Export with verbose output
stella airgap jobs export --tenant my-tenant -o bundle.json --verbose

# Export as JSON for automation
stella airgap jobs export --tenant my-tenant --json

Options:

• --tenant, -t - Tenant ID (defaults to “default”)
• --output, -o - Output file path
• --node - Export specific node only (default: current node)
• --sign - Sign bundle with DSSE
• --json - Output result as JSON
• --verbose - Enable verbose logging

Import Job Logs

Import a job sync bundle from air-gap transfer:

# Verify bundle without importing
stella airgap jobs import bundle.json --verify-only

# Import bundle
stella airgap jobs import bundle.json

# Force import despite validation issues
stella airgap jobs import bundle.json --force

# Import with JSON output for automation
stella airgap jobs import bundle.json --json

Options:

• bundle - Path to job sync bundle file (required)
• --verify-only - Only verify the bundle without importing
• --force - Force import even if validation fails
• --json - Output result as JSON
• --verbose - Enable verbose logging

List Available Bundles

List job sync bundles in a directory:

# List bundles in current directory
stella airgap jobs list

# List bundles in specific directory
stella airgap jobs list --source /path/to/bundles

# Output as JSON
stella airgap jobs list --json

Options:

• --source, -s - Source directory (default: current directory)
• --json - Output result as JSON
• --verbose - Enable verbose logging

Bundle Format

Job sync bundles are JSON files with the following structure:

{
  "bundleId": "guid",
  "tenantId": "string",
  "createdAt": "ISO8601",
  "createdByNodeId": "string",
  "manifestDigest": "sha256:hex",
  "signature": "base64 (optional)",
  "signedBy": "keyId (optional)",
  "jobLogs": [
    {
      "nodeId": "string",
      "lastHlc": "HLC timestamp string",
      "chainHead": "base64",
      "entries": [
        {
          "nodeId": "string",
          "tHlc": "HLC timestamp string",
          "jobId": "guid",
          "partitionKey": "string (optional)",
          "payload": "JSON string",
          "payloadHash": "base64",
          "prevLink": "base64 (null for first)",
          "link": "base64",
          "enqueuedAt": "ISO8601"
        }
      ]
    }
  ]
}

Validation

Bundle validation checks:

1. Manifest digest: Recomputes digest from job logs and compares
2. Chain integrity: Verifies each entry’s prev_link matches expected
3. Link verification: Recomputes links and verifies against stored values
4. Chain head: Verifies last entry link matches node’s chain head

Merge Algorithm

When importing bundles from multiple nodes:

1. Collect: Gather all entries from all node logs
2. Sort: Order by HLC total order (PhysicalTime, LogicalCounter, NodeId, JobId)
3. Deduplicate: Same JobId = same payload (drop later duplicates)
4. Recompute chain: Build unified chain from merged entries

This produces a deterministic ordering regardless of import sequence.

Conflict Resolution

Metrics

The following metrics are emitted:

Service Registration

To use job sync in your application:

// Register core services
services.AddAirGapSyncServices(nodeId: "my-node-id");

// Register file-based transport (for air-gap)
services.AddFileBasedJobSyncTransport();

// Or router-based transport (for connected scenarios)
services.AddRouterJobSyncTransport();

// Register sync service (requires ISyncSchedulerLogRepository)
services.AddAirGapSyncImportService();

Notes:

• File-based transport resolves bundle paths under the configured input/output roots and rejects traversal outside those roots.
• AddAirGapSyncServices registers FileBasedHlcStateStore by default. It writes hlc-state.json under FileBasedOfflineJobLogStoreOptions.DataDirectory so node HLC state survives restarts and stale timestamps cannot roll the node clock backward.
• Tests and specialized harnesses can still register a custom IHlcStateStore before calling AddAirGapSyncServices; the default registration uses TryAddSingleton.

Operational Runbook

Pre-Export Checklist

• [ ] Node has offline job logs to export
• [ ] Target path is writable
• [ ] Signing key available (if --sign used)

Pre-Import Checklist

• [ ] Bundle file accessible
• [ ] Bundle signature verified (if signed)
• [ ] Scheduler database accessible
• [ ] Sufficient disk space

Recovery Procedures

Chain validation failure:

1. Identify which entry has chain break
2. Check for data corruption in bundle
3. Re-export from source node if possible
4. Use --force only if data loss is acceptable

Duplicate conflict:

1. This is expected - duplicates are safely dropped
2. Check duplicate count in output
3. Verify merged jobs match expected count

Payload mismatch (same JobId, different payloads):

1. This indicates a bug - same idempotency key should produce same payload
2. Review job generation logic
3. Do not force import - fix root cause

See Also

• Air-Gap Operations
• Mirror Bundles
• Staleness and Time