Post

Lessons from Taming Clock Skew in Distributed Jobs

We ran into a strange issue: scheduled jobs running “twice” or not at all across nodes. The culprit? Clock skew.

In distributed systems, time is an illusion—but you still need some notion of coordination. What helped:

  • Using ntpd and tighter clock sync configurations
  • Avoiding “run at exactly X” semantics
  • Designing retries with logical idempotency

These bugs are rare, but they teach you humility. Distributed systems are always one leap second away from surprising you.

This post is licensed under CC BY 4.0 by the author.