記録 · Entry

Clock skew: why timestamps cannot order events across servers

Two servers never agree on the time, so the one running behind overwrites the one ahead. What each clock is for, and what to sequence with instead.

10 min read

Distributed SystemsBackendReliabilityTimekeepingConcurrency

Two application servers receive two updates to the same record, a hundred milliseconds apart. Each stamps its update with Date.now() and writes it. The storage layer keeps whichever timestamp is larger, because that is the newer one.

Server A's clock is a hundred and forty milliseconds ahead of server B's. The first update, on A, gets the larger stamp. The second update, the one the user actually made last, is on B, and it loses. The record now holds the older value, no error was raised, and every log line agrees that everything happened in the right order, because the logs used the same clocks.

This is not a rare race. It is what happens by default, every day, on every fleet where wall-clock time is used to decide which of two things came first. The fix is not a better clock. The fix is to stop asking clocks a question they cannot answer.

#Wall clocks are estimates, and they move backwards

The clock on a server is a crystal oscillator counting ticks, and every crystal runs slightly fast or slightly slow. Uncorrected, a typical one drifts by tens of parts per million, which is a few seconds a day. NTP corrects this by periodically comparing against a reference and adjusting.

The adjustment is the problem. When the offset is small, NTP slews: it speeds the clock up or slows it down slightly until it converges, which means for a while a second on this machine is not a second. When the offset is large, NTP steps: it sets the clock, forwards or backwards, in one jump. A machine that comes back from suspend, or a VM migrated between hosts, or a container on a host whose clock was just corrected, can see its clock move backwards by seconds.

A timestamp taken after that step is smaller than one taken before it. Two events on the same machine, in the correct order, with timestamps in the wrong order. And this is one machine. Across a fleet, even with NTP healthy, offsets of a few milliseconds are normal, tens of milliseconds are common, and a machine whose NTP daemon has quietly stopped can be seconds or minutes off with nothing reporting it.

The conclusion is not that clocks are bad. It is that a wall-clock timestamp is an estimate of when something happened, with an unknown error, and an estimate with an unknown error cannot be used to decide which of two nearby things happened first.

#Durations need the other clock

There are two clocks in every machine, and the first mistake is using the same one for both jobs.

The wall clock, Date.now(), SystemTime::now(), CLOCK_REALTIME, tells you what time it is. It can be stepped.

The monotonic clock, performance.now(), process.hrtime.bigint(), Instant::now(), CLOCK_MONOTONIC, tells you how much time has passed since some arbitrary point. It only goes forward. It has no meaning across machines and no meaning across reboots, and it is the only clock that can measure a duration.

const started = Date.now()
await work()
const elapsed = Date.now() - started

elapsed here can be negative. It can also be an hour, if NTP stepped the clock during work(). Every timeout, every latency measurement, every "has this lease expired" check computed from the wall clock is exposed to that.

const started = performance.now()
await work()
const elapsed = performance.now() - started

This one is correct. The rule is short: wall clock for when, monotonic clock for how long. A deadline sent across the network is a duration, "you have 800 milliseconds left", precisely because the receiver's wall clock cannot be trusted to agree with the sender's about an absolute time.

#Last-write-wins is a bet on the clocks

Any system that resolves concurrent writes by comparing timestamps has made an assumption that the clocks agree to within the interval between writes. For writes minutes apart, the assumption holds. For writes from two servers inside the same second, it does not, and that is the case that matters, because that is the case where two writes were concurrent in the first place.

Cassandra's conflict resolution is timestamp-based last-write-wins, and its documentation is honest that clock synchronisation across nodes is a correctness requirement, not an operational nicety. Any caching layer that stores updated_at and keeps the larger one, any sync protocol that sends "my version is from 14:03:22.114", any deduplication that assumes a later timestamp means a later event, is making the same bet.

The bet is not always wrong. If the cost of losing it is a stale field in a profile page, it may be fine. If the cost is a payment applied in the wrong order, it is not, and the design has to find order somewhere other than the clock.

#Order comes from a single writer

The reliable way to know that B came after A is for the same thing to have seen both, in that order, and to have said so.

A database sequence does this. nextval() is executed by the database, on one server, and returns strictly increasing values in the order the calls arrived. Two application servers with wildly different clocks get numbers that reflect the order the database saw them, which is the only order that exists.

A Kafka partition does this. Every record gets the next offset, assigned by the leader broker, in the order the records were appended. Within a partition the order is total and it is not derived from any clock. Across partitions there is no order at all, which is why the partition key decides what "in order" means for a consumer.

A row's version column does this. UPDATE ... SET version = version + 1 WHERE id = $1 AND version = $2 succeeds for exactly one of two concurrent writers, and the loser is told, rather than silently overwritten. Optimistic concurrency is ordering by the single writer that is the row itself.

Every one of these gives up something: throughput through one sequence, ordering across partitions, the need to retry a lost update. That is the price of an order that is real. A timestamp comparison gives up nothing and is not real.

#Sortable ids are not orderable ids

UUIDv7 puts a millisecond timestamp in the high bits, so ids sort by roughly the time they were created. This is excellent for index locality, because inserts land near each other instead of scattering. It is a trap if you read it as an ordering.

Two servers generating v7 ids for two events a few milliseconds apart will produce ids whose order matches their clocks, not the events. Within one process, the spec's monotonic counter makes consecutive ids increase even inside the same millisecond. Across processes there is no such guarantee and the skew between machines is larger than the gap between events. Snowflake ids have the same property for the same reason: a node id and a sequence make them unique, the timestamp makes them roughly sortable, and neither makes them a statement about which of two events on different nodes happened first.

Use them for what they are: unique, and kind to the index. Not as a sequence.

#When you need order without a single writer

Sometimes there is no single point that sees everything, and the system still needs to know what happened before what. That question has a fifty-year-old answer, and it does not involve reading a clock.

A Lamport clock is a counter. Each process increments it on every event, attaches it to every message, and on receiving a message sets its own counter to one more than the larger of its own and the received value. The result is that if A caused B, A's counter is smaller than B's. The reverse does not hold: two events with counters 5 and 7 may be unrelated, and the counters say nothing about which happened first in wall time. What it gives is exactly what a timestamp cannot: a guarantee that causality is never reversed.

A vector clock extends this to detect concurrency. Each process keeps a counter per process, and two events are concurrent when neither vector dominates the other. That is the signal a system needs to say "these two writes conflict, and no order between them exists," rather than picking one with a coin toss dressed as a timestamp. It is what Dynamo-style stores used, and it is why they surface conflicts to the application instead of resolving them silently.

A hybrid logical clock is the practical compromise. It carries a physical timestamp so values are close to wall time and readable by humans, plus a logical counter that increments when the physical component would have gone backwards or collided. Ordering follows causality like a Lamport clock, and values stay within a bounded distance of real time. CockroachDB and others use HLCs for exactly this reason: timestamps that mean something to a person and still never reverse a cause and its effect.

#What Spanner did, and why you probably cannot

Google's Spanner does order transactions across a global fleet by wall-clock time, and it can do so because of a piece of hardware. TrueTime returns not a timestamp but an interval, [earliest, latest], backed by GPS receivers and atomic clocks in every data centre, with the interval width, typically a few milliseconds, being the honest uncertainty. A transaction that commits at time T then waits until now.earliest is past T before releasing its result. That wait is what makes the timestamp trustworthy: nobody can observe the commit until every clock in the system agrees it is in the past.

Two things follow. First, even with atomic clocks, correctness required admitting the uncertainty and waiting it out, which is the opposite of comparing two Date.now() values and hoping. Second, cloud providers now sell clock services with microsecond accuracy, and they are worth using, and they still do not change the rule. A better clock narrows the interval. It does not make the interval zero, and an ordering decision made inside the interval is still a guess.

#What to do on Monday

Measure durations with the monotonic clock, everywhere, and treat any Date.now() subtraction as a bug. Timeouts, leases, retry backoff, rate limit windows: all durations.

Store wall-clock timestamps for humans and for approximate queries. "Orders from last Tuesday" is fine. "This write is newer than that write" is not.

Sequence with a single writer wherever one exists: a database sequence, a partition offset, a version column. Use optimistic concurrency so that concurrent writes fail loudly instead of quietly losing one.

Where no single writer exists, carry a logical clock and let the application see conflicts. Do not let the storage layer resolve them by comparing numbers that two machines wrote with two different ideas of what time it was.

And run NTP monitoring as a first-class alert. A server that drifts a second is a server that will, sooner or later, be the one whose write wins when it should have lost.

#The one-line summary

A wall clock tells you roughly when, a monotonic clock tells you exactly how long, and neither tells you which of two events on two machines came first. Order comes from a single writer or a logical clock, never from comparing timestamps.

#References

Related entries