---
url: >-
  https://cubesandbox.com/guide/troubleshooting/node-heartbeat-expired-clock-skew.md
---

# Compute nodes report HeartbeatExpired while still online (clock skew)

## Symptom

A compute node shows `Healthy=false` with `UnhealthyReason=HeartbeatExpired` in `cubeopscli node ls --json` (the WebUI `/nodes` page only marks the node with an amber dot — it renders no status text and does not surface the reason), yet the node is actually up — `cubelet` is running and posting heartbeats. Sandbox creation that should land on this node fails with `no ready replica` / `no more resource`, because the scheduler filters out `HeartbeatExpired` nodes.

The key tell: the node's `MetaDataUpdateAt` timestamp keeps advancing (the heartbeat IS arriving), but the node is still judged expired.

## Environment

* CubeSandbox v0.7.x, multi-node deployment (control node and compute node(s) on separate hosts)
* Compute node clock is not synchronized to a common time source (NTP/chrony)
* Related components: `cubelet` (heartbeat reporter), `cube-ops` (health evaluator)

## Root Cause

Cubelet reports a status heartbeat about once per second to `cube-ops` (`POST /internal/v1/node-agent/nodes/<node-id>/status`; `node_status_update_frequency` defaults to `1s` in `Cubelet/config/config.toml`; `<node-id>` is the node's identifier — under the K8s chart it is the node name, under the one-click/bare-metal path it is typically the configured node ID), and the `HeartbeatTime` field is stamped with the **compute node's local wall clock**. `cube-ops` evaluates health with its **own** local wall clock as `now`:

```go
// CubeOps/internal/nodemanagement/service/health.go
func EvaluateHealth(reportedReady bool, heartbeatTime, now time.Time, timeout time.Duration) HealthStatus {
    if heartbeatTime.IsZero() || now.Sub(heartbeatTime) > timeout {
        return HealthStatus{Healthy: false, UnhealthyReason: model.ReasonHeartbeatExpired}
    }
    ...
}

// MetadataTimeout returns the cubelet heartbeat expiry window (15s)
func MetadataTimeout() time.Duration { return 15 * time.Second }
```

The window is 15s. If the compute node's clock is more than 15s behind the control node's clock, the `HeartbeatTime` it stamps is more than 15s in the past from `cube-ops`'s perspective — so every fresh heartbeat is already "expired" on arrival. The heartbeat traffic is fine; only the two clocks disagree.

This is not a code bug — it's the standard distributed-systems assumption that all nodes share a common time reference. Compute nodes whose `chrony` has no server configured (`Stratum=0`, `Reference ID=00000000`) drift freely, and VM/PVM compute nodes are especially prone to drift.

## Diagnosis

1. Check the node state and reason. The default `node ls` table only prints `NODE_ID / NODE_IP / INSTANCE_TYPE / ZONE / CPU_TYPE / HEALTHY / SCHEDULING_DISABLED / HOST_STATUS` — it does **not** show `UnhealthyReason` or the heartbeat timestamp, so use `--json` (full fields, shows both) or `--score-only` (switches to a score/time view that includes `METADATA_UPDATE` but drops `HEALTHY`):

   ```bash
   cubeopscli node ls --json          # full fields: Healthy, UnhealthyReason, MetaDataUpdateAt
   # or, to watch the heartbeat timestamp advance (this view drops HEALTHY/UnhealthyReason):
   cubeopscli node ls --score-only    # NODE_ID / SCORE / METRIC_UPDATE / METRIC_LOCAL_UPDATE / METADATA_UPDATE
   # Look for: Healthy=false, UnhealthyReason=HeartbeatExpired,
   # and a MetaDataUpdateAt that is still ADVANCING (it may look stale
   # by the skew amount, since it is populated from the compute node's
   # own HeartbeatTime — sample it twice to confirm it moves, rather
   # than judging "recent" by the control node's clock).
   # (--json is the mode that shows both the reason and the timestamp together)
   ```

2. Distinguish clock skew from a genuinely offline node. **If `MetaDataUpdateAt` stopped advancing days ago**, `cubelet` is actually stopped/dead — that is a real offline node, not this issue. **If `MetaDataUpdateAt` keeps updating** but the node is still `HeartbeatExpired`, the clocks disagree.

   Note the comparison is **relative**: `EvaluateHealth` checks `now.Sub(heartbeatTime)` using `cube-ops`'s wall clock (`now = time.Now()` on the control node), so it trips only when the heartbeat time is in the **past** relative to the control node's clock — i.e. when the compute node's clock is **behind** the control node's. Either end failing to sync can cause that: the compute node drifting slow, or the control node jumping ahead (its own chrony unsynced, or a VM pause/resume on the control node). The symptom itself is one-directional (a compute node that is fast stays healthy), but either clock can be the source of the relative skew, so check **both** — the compute node first, and if it looks fine, the **control** node too (`chronyc tracking` on each).

3. On the compute node, check whether chrony is synchronized:

   ```bash
   chronyc tracking
   # Stratum: 0  and  Reference ID: 00000000  => NOT synchronized, free-running
   # If the compute node looks fine here but the node is still HeartbeatExpired,
   # run the same check on the CONTROL node — it may have jumped ahead.
   ```

4. Measure the offset between the compute node and the control node. The simplest way is to compare wall clocks:

   ```bash
   # On the compute node, note the time, then SSH to the control node and compare:
   date +%s   # on compute node
   ssh <control-node-ip> date +%s   # on control node
   # The compute node must be BEHIND the control node: EvaluateHealth checks
   # `now.Sub(heartbeatTime) > timeout`, so only a slow compute clock reproduces
   # HeartbeatExpired. A compute node that is ahead stays healthy (the delta is
   # negative). A difference larger than 15s in that direction reproduces the
   # false HeartbeatExpired.
   ```

   Or, for a one-shot offset measurement against a specific NTP source, run this **on the compute node** (it prints `System clock ... wrong by <N> seconds` and does not modify the clock). The target must actually serve NTP — use the same `<your-ntp-server>` the control node syncs to (see Resolution step 1); if the control node is not an NTP server, point this at a real NTP source instead:

   ```bash
   chronyd -Q 'server <your-ntp-server> iburst'   # one-shot probe; prints the offset vs. that server
   ```

## Resolution

Both ends must be synchronized to a common time source — the comparison is relative, so fixing only the compute node does not recover the node if it was the control node's clock that jumped ahead. First verify both clocks are synced (diagnosis step 3 on each node), then synchronize whichever end is drifting.

1. Confirm the time source. If your control node runs chrony as a local NTP server (e.g. `local stratum 10` + `allow`), use it; otherwise use a public NTP pool or your organization's NTP server. Pick exactly the source the control node is synced to, so both clocks agree. If the control node itself is not synced (its `chronyc tracking` shows `Stratum=0`), fix the control node first — pointing the compute node at a source the control node does not use will not make the clocks agree.

2. Configure chrony on the compute node. The config path and service name differ by distribution:

   ```bash
   # ── RPM family (OpenCloudOS / TencentOS / RHEL / CentOS): /etc/chrony.conf, service chronyd ──
   sudo sed -i '/^server /d; /^pool /d' /etc/chrony.conf
   echo 'pool pool.ntp.org iburst' | sudo tee -a /etc/chrony.conf
   # Or, for an internal NTP server:
   # echo 'server <your-ntp-server> iburst' | sudo tee -a /etc/chrony.conf
   sudo systemctl restart chronyd

   # ── Debian / Ubuntu: /etc/chrony/chrony.conf, service chrony ──
   # (Ubuntu defaults to systemd-timesyncd; install chrony first if it is missing.)
   # sudo apt-get install -y chrony
   sudo sed -i '/^server /d; /^pool /d' /etc/chrony/chrony.conf
   echo 'pool pool.ntp.org iburst' | sudo tee -a /etc/chrony/chrony.conf
   # Or, for an internal NTP server:
   # echo 'server <your-ntp-server> iburst' | sudo tee -a /etc/chrony/chrony.conf
   sudo systemctl restart chrony
   ```

   > Use `pool` for a name that resolves to multiple addresses (like `pool.ntp.org`); `server` pins a single resolved address at startup. Use `server` for a single internal NTP host.

3. Restart chrony and verify the offset collapses to near zero (RPM family: `chronyd`; Debian/Ubuntu: `chrony`). After the restart, if the offset is large, chrony corrects it by **slewing** (gradually, default max slew rate ~8.3%/s) unless told to step — a skew beyond 15s can take much longer than one heartbeat cycle to close, and a skew of minutes/hours can take hours. Step the clock immediately if the node needs to recover fast:

   ```bash
   sudo systemctl restart chronyd    # RPM family — or: sudo systemctl restart chrony   (Debian/Ubuntu)
   chronyc tracking   # Stratum should now be non-zero
   sudo chronyc makestep   # step the clock now instead of slewing; safe to repeat if offset remains
   chronyc tracking   # offset should now be ~0
   # (or add `makestep 1.0 3` to the chrony config so the first 3 updates past 1s step automatically)
   ```

4. Once the clocks agree, the node turns healthy once the next heartbeat is evaluated — verify:

   ```bash
   cubeopscli node ls   # Healthy=True for the fixed node
   ```

No `cubelet`/`cube-ops` restart or config change is needed; once the clocks agree, the existing heartbeat stream is evaluated as fresh.

## References

* Heartbeat evaluation: `CubeOps/internal/nodemanagement/service/health.go` (`EvaluateHealth`, `MetadataTimeout`)
* Heartbeat reporting: `Cubelet/pkg/cubelet/node_status.go` (`HeartbeatTime: kl.clock.Now()`)
* Related: #455 (stale node health handling — this is a different failure mode: the heartbeat arrives on time, but the clocks disagree)