HLC Troubleshooting Runbook

Version: 1.2.0
Sprint: SPRINT_20260105_002_004_BE
Last Updated: 2026-05-31

Audience: On-call SREs and the Scheduler team troubleshooting Hybrid Logical Clock (HLC) based queue ordering in the Stella Ops JobEngine scheduler. This runbook covers chain-verification failures, clock skew, air-gap merge conflicts, and node-identity collisions.

Status note (reconciled 2026-05-31 against src/JobEngine and src/__Libraries/StellaOps.HybridLogicalClock): HLC ordering is implemented as a library in StellaOps.Scheduler.Queue (see HlcSchedulerOptions in src/JobEngine/StellaOps.Scheduler.__Libraries/StellaOps.Scheduler.Queue/Options/HlcSchedulerOptions.cs and the 002_hlc_queue_chain.sql migration). However, several monitoring hooks this runbook originally assumed do not exist yet in code and are flagged inline below:

• The HLC queue is not wired into any running host. Neither DI entry point — ServiceCollectionExtensions.AddHlcSchedulerQueue() (registers the Services/* implementations) nor HlcSchedulerServiceCollectionExtensions.AddHlcSchedulerServices() (registers the Hlc/* implementations) — is called anywhere outside the library itself and its test project. The Scheduler WebService (StellaOps.Scheduler.WebService/Program.cs) does not register HLC ordering, and EnableHlcOrdering defaults to false. Treat this entire runbook as forward-looking / roadmap for a feature whose enqueue/dequeue/verify code paths are not exercised at runtime in the shipped Scheduler. The schema (migration 002_hlc_queue_chain.sql) and the core HLC library are present.
• No Prometheus alert rules are shipped for any of the Hlc* alert names referenced here (no rule appears under devops/telemetry/). Treat alert names as descriptive labels for the condition, not as wired alerts.
• No /health/hlc or /config HTTP endpoint exists. The Scheduler WebService maps only /healthz and /readyz (plus a build-info endpoint and the domain job/schedule/run endpoints). There is no /health/liveness route on the Scheduler service.
• The HLC/scheduler metric instruments below are declared but never recorded to: no code path calls the RecordEnqueue/RecordDequeue/RecordChainVerification/RecordSnapshotCreated (etc.) methods, so the counters/histograms emit nothing at runtime even when the feature would be enabled. Of the air-gap sync metrics, only airgap_bundle_size_bytes has a live call site (RecordBundleSize); the rest (including airgap_merge_conflicts_total and airgap_sync_duration_seconds) are also declared-but-unrecorded. See the Metrics reference for the full declared-vs-emitted breakdown.
• Several legacy metric names referenced by older docs (hlc_clock_skew_rejections_total, hlc_physical_time_offset_seconds, hlc_ticks_total, scheduler_jobs_received_total) do not exist at all in src/.

Table of Contents

1. Chain Verification Failure
2. Clock Skew Issues
3. Time Offset Drift
4. Merge Conflicts
5. Slow Air-Gap Sync
6. No Enqueues
7. Batch Snapshot Failures
8. Duplicate Node ID

1. Chain Verification Failure

Symptoms

• Condition label: HlcChainVerificationFailure (NOT a shipped Prometheus alert — see status note)
• Metric: scheduler_chain_verification_failures_total (declared on the StellaOps.Scheduler.HlcQueue meter but not recorded to by current code — see status note and the Metrics reference; do not expect this counter to move at runtime today)
• Log (per-issue): the exact strings depend on which SchedulerChainVerifier is wired — there are two implementations with different messages:
  • Hlc/SchedulerChainVerifier(registered by AddHlcSchedulerServices): Stored link doesn't match computed. Stored=..., Computed=... (LinkMismatch) or Expected ..., got ... (PrevLinkMismatch), followed by the summary line Chain verification complete. TenantId=..., ..., IsValid=false, IssueCount=N.
  • Services/SchedulerChainVerifier(registered by AddHlcSchedulerQueue): Stored link doesn't match computed link (LinkMismatch), PrevLink doesn't match previous entry's link (PrevLinkMismatch), plus an InvalidPayloadHash/InvalidLinkLength/HlcOrderViolation issue set the other does not raise; summary line Chain verification for tenant {TenantId}: VALID|INVALID, {N} entries, {M} issues.
  • This duplication is a known code smell, not a documented contract. Confirm which type the deployment registers before pattern-matching log strings.

Severity

Critical - Indicates potential data tampering or corruption.

Investigation Steps

1. Identify the affected chain segment:

   SELECT 
       job_id, 
       t_hlc, 
       encode(prev_link, 'hex') as prev_link,
       encode(link, 'hex') as link,
       created_at
   FROM scheduler.scheduler_log
   WHERE tenant_id = '<tenant_id>'
   ORDER BY t_hlc DESC
   LIMIT 100;
   

2. Find the break point:

   WITH chain AS (
       SELECT 
           job_id,
           t_hlc,
           prev_link,
           link,
           LAG(link) OVER (ORDER BY t_hlc) as expected_prev
       FROM scheduler.scheduler_log
       WHERE tenant_id = '<tenant_id>'
   )
   SELECT * FROM chain
   WHERE prev_link IS DISTINCT FROM expected_prev
   ORDER BY t_hlc;
   

3. Check for unauthorized modifications:

   • Review database audit logs
   • Check for direct SQL updates bypassing the application

4. Verify chain head consistency:

   SELECT * FROM scheduler.chain_heads
   WHERE tenant_id = '<tenant_id>';
   

Resolution

If corruption is isolated:

1. Mark affected jobs for re-processing
2. Rebuild chain from the last valid point
3. Update chain head

If tampering is suspected:

1. Escalate to Security team immediately
2. Preserve all logs and database state
3. Initiate incident response procedure

2. Clock Skew Issues

Symptoms

• Condition label: HlcClockSkewExceedsTolerance (NOT a shipped Prometheus alert — see status note)
• Metric: hlc_clock_skew_rejections_total is NOT IMPLEMENTED. There is no dedicated skew-rejection counter; clock-skew rejection currently surfaces only as an error log and an HlcClockSkewException thrown from HybridLogicalClock.Receive().
• Log (from StellaOps.HybridLogicalClock.HybridLogicalClock.Receive): Clock skew of {Skew} from node {RemoteNode} exceeds threshold {MaxSkew} (logged at Error; {Skew}/{MaxSkew} are formatted as TimeSpan values, not raw milliseconds)

Severity

Critical - Can cause job ordering inconsistencies.

Investigation Steps

1. Check NTP synchronization:

   # On affected node
   timedatectl status
   ntpq -p
   chronyc tracking  # if using chrony
   

2. Verify time sources:

   ntpq -pn
   

3. Check for leap second issues:

   dmesg | grep -i leap
   

4. Compare with other nodes:

   for node in node-1 node-2 node-3; do
     echo "$node: $(ssh $node date +%s.%N)"
   done
   

Resolution

1. Restart NTP client:

   sudo systemctl restart chronyd  # or ntpd
   

2. Force time sync:

   sudo chronyc makestep
   

3. Temporarily increase tolerance (emergency only):

   Scheduler:
     HlcOrdering:
       MaxClockSkewMs: 60000  # Increase from default 1000 (max allowed: 60000)
   

   Config section is Scheduler:HlcOrdering (constant HlcSchedulerOptions.SectionName), the key is MaxClockSkewMs (default 1000, range [0, 60000]). The skew threshold actually enforced at runtime is the maxClockSkew passed when constructing HybridLogicalClock (library default 1 minute); confirm both are aligned for the deployment.

4. Restart affected service to reset HLC state.

3. Time Offset Drift

Symptoms

• Condition label: HlcPhysicalTimeOffset (NOT a shipped Prometheus alert — see status note)
• Metric: hlc_physical_time_offset_seconds is NOT IMPLEMENTED. No physical-time-offset gauge is emitted by current code (the archived design doc docs-archive/modules/scheduler/hlc-ordering.md lists it as hlc_physical_offset_seconds, but neither name exists in src/). Detect drift indirectly: a persistently high HLC logical counter (the trailing -NNNNNN segment of t_hlc) indicates the wall clock is lagging and the logical counter is absorbing the gap.

Severity

Warning - May cause timestamp anomalies in diagnostics.

Investigation Steps

1. Inspect recent HLC timestamps directly (the logical counter is the last --delimited segment of t_hlc):

   SELECT job_id, t_hlc, created_at
   FROM scheduler.scheduler_log
   WHERE tenant_id = '<tenant_id>'
   ORDER BY t_hlc DESC
   LIMIT 50;
   

2. Review HLC state: there is no /health/hlc endpoint. The Scheduler WebService maps only /healthz and /readyz (there is no /health/liveness route on this service). Inspect the scheduler.scheduler_log / scheduler.chain_heads tables (above and in §1) for current ordering state.

3. Check for high logical counter: If the logical counter (the t_hlc tail segment) is very high, it indicates frequent same-millisecond events or a wall clock that is not advancing.

Resolution

Usually self-correcting as wall clock advances. If persistent:

1. Review job submission rate
2. Consider horizontal scaling to distribute load

4. Merge Conflicts

Symptoms

• Condition label: HlcMergeConflictRateHigh (NOT a shipped Prometheus alert — see status note)
• Metric: airgap_merge_conflicts_total is declared on the StellaOps.AirGap.Sync meter (tagged conflict_type) but its recorder (RecordMergeConflict) currently has no call site, so the counter does not move at runtime today (see status note). When wired, tag value would be DuplicateTimestamp or PayloadMismatch.
• Conflict types are DuplicateTimestamp and PayloadMismatch (the ConflictType enum in StellaOps.AirGap.Sync.Models.ConflictResolution)

Severity

Warning - May indicate duplicate job submissions or clock issues on offline nodes.

Investigation Steps

1. Identify conflict types (tag value is DuplicateTimestamp or PayloadMismatch):

   sum by (conflict_type) (airgap_merge_conflicts_total)
   

2. Review merge logs:

   grep "merge conflict" /var/log/stellaops/scheduler.log | tail -100
   

3. Check offline node clocks:

   • Were offline nodes synchronized before disconnection?
   • How long were nodes offline?

Resolution

1. For duplicate jobs: Use idempotency keys to prevent duplicates
2. For payload conflicts: Review job submission logic
3. For ordering conflicts: Verify NTP on all nodes before disconnection

5. Slow Air-Gap Sync

Symptoms

• Condition label: HlcSyncDurationHigh (NOT a shipped Prometheus alert — see status note)
• Metric: airgap_sync_duration_seconds (histogram declared on the StellaOps.AirGap.Sync meter) p95 > 30s — but its recorder (RecordSyncDuration) has no call site in current code, so no samples are produced today (see status note). airgap_bundle_size_bytes (used in the investigation step below) is recorded (RecordBundleSize) and is the one air-gap sync histogram with a live emission path.

Severity

Warning - Delays job processing.

Investigation Steps

1. Check bundle sizes:

   histogram_quantile(0.95, airgap_bundle_size_bytes_bucket)
   

2. Check database performance:

   SELECT * FROM pg_stat_activity
   WHERE state = 'active' AND query LIKE '%scheduler_log%';
   

3. Review index usage:

   EXPLAIN ANALYZE
   SELECT * FROM scheduler.scheduler_log
   WHERE tenant_id = '<tenant>'
   ORDER BY t_hlc
   LIMIT 1000;
   

Resolution

1. Chunk large bundles: Split bundles > 10K entries
2. Optimize database: Ensure indexes are used
3. Increase resources: Scale up database if needed

6. No Enqueues

Symptoms

• Condition label: HlcEnqueueRateZero (NOT a shipped Prometheus alert — see status note)
• No jobs appearing in HLC queue. Note: scheduler_hlc_enqueues_total is declared but never recorded (see status note), so it is always flat today regardless of activity — it cannot distinguish “no enqueues” from “feature not wired”. Verify enqueue activity by querying scheduler.scheduler_log row counts instead.

Severity

Info - May be expected or indicate misconfiguration.

Investigation Steps

1. Check if HLC ordering is enabled: there is no /config HTTP endpoint. Confirm the effective configuration from the service’s bound Scheduler:HlcOrdering section (appsettings / environment overrides) — the relevant flag is EnableHlcOrdering (default false).

2. Verify enqueue activity (there is no scheduler_jobs_received_total metric, and scheduler_hlc_enqueues_total is declared-but-unrecorded — see status note). Until the enqueue path records metrics, confirm activity directly from the table:

   SELECT COUNT(*) AS recent_enqueues
   FROM scheduler.scheduler_log
   WHERE tenant_id = '<tenant_id>'
     AND created_at > NOW() - INTERVAL '5 minutes';
   

   If/when the metric is wired, the equivalent PromQL would be rate(scheduler_hlc_enqueues_total[5m]).

3. Check for errors:

   grep -i "hlc\|enqueue" /var/log/stellaops/scheduler.log | grep -i error
   

Resolution

1. If HLC should be enabled:

   Scheduler:
     HlcOrdering:
       EnableHlcOrdering: true
   

2. If dual-write mode is needed (the option is named EnableDualWrite, not DualWriteMode):

   Scheduler:
     HlcOrdering:
       EnableDualWrite: true
   

   Migration phases (per HlcSchedulerOptions and the queue MIGRATION_GUIDE.md): Phase 1 EnableDualWrite=true, EnableHlcOrdering=false; Phase 2 EnableDualWrite=true, EnableHlcOrdering=true; Phase 3 EnableDualWrite=false, EnableHlcOrdering=true.

7. Batch Snapshot Failures

Symptoms

• Condition label: HlcBatchSnapshotFailures (NOT a shipped Prometheus alert — see status note)
• Missing DSSE-signed batch proofs. The scheduler_batch_snapshots_total counter is declared-but-unrecorded (see status note), so it is always flat today; the reliable signal is the table itself — rows in scheduler.batch_snapshot with NULL signed_by/signature when signing was expected. (DB constraint chk_signature_requires_signer enforces that signed_by and signature are either both NULL or both set, so an unsigned row has both NULL.)

Severity

Warning - Audit proofs may be incomplete.

Implementation note: Batch-snapshot signing is HMAC-based, not certificate-based, and there is no stella signer CLI for it. Signing is performed in-process by BatchSnapshotDsseSigner and configured via BatchSnapshotDsseOptions, which are bound from the configuration section Scheduler:Queue:Hlc:DsseSigning(note: this is a different section from Scheduler:HlcOrdering that holds the other HLC options, and only AddHlcSchedulerServicesWithDsseSigning(configuration) binds it). The options are: Mode ("hmac" or "none", default "none"), SecretBase64 (the HMAC secret, required when Mode="hmac"), KeyId (default scheduler-batch-snapshot), and PayloadType (default application/vnd.stellaops.scheduler.batch-snapshot+json). Signing is enabled iff Mode equals "hmac" (case-insensitive). The MAC routes through the central ICryptoHmac abstraction, so the algorithm follows the active compliance profile (HMAC-SHA256 on world/fips/kcmvp/eidas, HMAC-GOST3411 on gost, HMAC-SM3 on sm). Whether snapshots are taken/signed at all is gated by HlcSchedulerOptions.SignBatchSnapshots and BatchSnapshotIntervalSeconds (in Scheduler:HlcOrdering).

Investigation Steps

1. Check signing configuration: verify Scheduler:Queue:Hlc:DsseSigning:Mode is "hmac" and Scheduler:Queue:Hlc:DsseSigning:SecretBase64 is set (a missing secret throws InvalidOperationException: "HMAC signing mode requires SecretBase64 to be configured", and an invalid Base64 secret throws InvalidOperationException: "SecretBase64 is not valid Base64" at sign time). There is no stella signer status command for this signer.

2. Verify the feature is enabled: there is no /config HTTP endpoint. Confirm Scheduler:HlcOrdering:SignBatchSnapshots=true and a non-zero BatchSnapshotIntervalSeconds in the bound configuration.

3. Check database connectivity:

   SELECT 1;  -- Simple connectivity test
   

Resolution

1. Verify SecretBase64 is configured and is valid Base64 (an invalid value is logged and treated as a verification/signing failure).
2. Confirm the active compliance profile matches the algorithm used to produce earlier signatures (a profile switch changes the HMAC algorithm).
3. Verify database permissions for the scheduler.batch_snapshot table (note: the table is singular — scheduler.batch_snapshot, not batch_snapshots).

8. Duplicate Node ID

Symptoms

• Condition label: HlcDuplicateNodeId (NOT a shipped Prometheus alert — see status note)
• Multiple instances with the same NodeId

Severity

Critical - Will cause chain corruption.

Investigation Steps

1. Identify affected instances: the hlc_ticks_total metric referenced previously is NOT IMPLEMENTED — no per-node tick counter is emitted. Detect collisions another way:

   • Compare the NodeId embedded in t_hlc (physicalTime13-NodeId-counter6) across nodes by querying scheduler.scheduler_log. Caveat: the NodeId segment may itself contain hyphens (the parser is ^(\d{13})-(.+)-(\d{6})$, so e.g. scheduler-east-1 is a valid NodeId), so split_part(t_hlc,'-',2) returns only the first token of a multi-hyphen NodeId. Recover the full NodeId by stripping the leading 13-digit physical time and trailing 6-digit counter instead:

     SELECT
         substring(t_hlc FROM 15 FOR length(t_hlc) - 21) AS node_id,  -- drop 13-digit time + '-' prefix and '-' + 6-digit counter suffix
         COUNT(*)
     FROM scheduler.scheduler_log
     WHERE tenant_id = '<tenant_id>'
     GROUP BY 1 ORDER BY 2 DESC;
     

   • Cross-check the configured NodeId on each instance (step 2). NodeId defaults to Environment.MachineName when unset.

2. Check node ID configuration (option Scheduler:HlcOrdering:NodeId):

   # On each instance, inspect the bound Scheduler:HlcOrdering section
   grep -ri "NodeId" /etc/stellaops/
   

Resolution

Immediate action required:

1. Stop one of the duplicate instances
2. Reconfigure with unique node ID
3. Restart and verify
4. Check chain integrity for affected time period

Metrics reference

The metric names below are declared by current code, with their owning meter. Important: declared ≠ emitted. A meter instrument only produces samples when something calls its Record* method. As of this reconciliation, almost none of the HLC/scheduler instruments have a recording call site (the HLC queue services do not invoke either metrics class), and on the air-gap side only airgap_bundle_size_bytes is recorded. The “Emitted?” column reflects whether a live call site exists today. The legacy names called out as “NOT IMPLEMENTED” do not exist at all.

Meter StellaOps.Scheduler.HlcQueue (HlcSchedulerMetrics in StellaOps.Scheduler.Queue/Metrics)

A second, separately-registered metrics type also exists at StellaOps.Scheduler.Queue.Hlc.HlcSchedulerMetrics (a static class, meter StellaOps.Scheduler.Hlc) declaring overlapping and divergent names: scheduler_hlc_enqueues_total, scheduler_hlc_enqueue_deduplicated_total, scheduler_hlc_enqueue_duration_seconds, scheduler_hlc_dequeues_total, scheduler_hlc_dequeued_entries_total, scheduler_chain_verifications_total (tag result = valid/invalid, not success/failure), scheduler_chain_verification_issues_total, scheduler_chain_entries_verified_total, scheduler_batch_snapshots_created_total, scheduler_batch_snapshots_signed_total, and scheduler_batch_snapshot_verifications_total. These instruments are likewise not recorded to by any caller. Confirm which type (if any) the deployed Scheduler wires before building dashboards. This duplication is a known code smell, not a documented contract.

Meter StellaOps.AirGap.Sync (AirGapSyncMetrics)

Not implemented (referenced by older docs but absent from src/)

• hlc_ticks_total, hlc_clock_skew_rejections_total, hlc_physical_time_offset_seconds (a.k.a. hlc_physical_offset_seconds), scheduler_chain_entries_total, scheduler_jobs_received_total — none of these exact names exist anywhere in src/. (Note: a different name, scheduler_chain_entries_verified_total, is declared by the StellaOps.Scheduler.Hlc meter — see the duplication note above — but it too has no recording call site.)
• The Grafana dashboard devops/observability/grafana/hlc-queue-metrics.json cited by docs-archive/modules/scheduler/hlc-ordering.md does not exist at that path in this repo (the only files under devops/observability/grafana/ are canary.json and provisioning/).

Escalation Matrix

Revision History