One of my favorite services to run is NTP. The math that makes it work is
elegant, graphing NTP’s performance produces beautiful graphs, and NTP is
usually a low maintenance service. Most importantly, accurate time
synchronization is crucial to every day IT functions. This should be something
in every Operation Engineer’s tool bag.
Yet, its always difficult – socially – to make changes to an NTP
infrastructure. Every client I’ve worked with has been hesitant to
allow changes to their NTP configuration. Many assume that NTP is
“simple” and “working” why should it be changed?
Don’t assume that NTP is “simple” and ignore it. Like everything else, one
needs a good understanding of how it works to have accurate time
synchronization. Here are some tips for running an NTP infrastructure
that maintains accurate synchronization and works on all of your machines
– including those often drifting virtual machines. At the bottom
you’ll find an example NTP configuration.
UTC Is Your Friend
The first tip for setting up your infrastructure to have reasonably
accurate time is to set your BIOS clock to UTC. UTC doesn’t have
Daylight Savings of other weird time changes. Do not keep your BIOS
clock in the local time.
Hierarchy of NTP
NTP servers build a hierarchy and the level you are on in that hierarchy is
called your stratum. The smaller the stratum, the more accurate your
synchronization should be. However, there are many thousands (or more) of
these hierarchies. You can use more than one (and you should). They may
overlap upstream from your servers. This is used to create a large amount of
redundancy. But, it can also cause hidden single points of failure. Do some
work to identify your upstream sources.
For one or a handful of servers, VMs, or workstations directly connected to the
Internet, using NTP sources from a pool is best. Your distro most likely
has a good starting point for your NTP configuration in
For more machines or machines on a backend network you want to setup your own
bit of NTP hierarchy. Normal machines will synchronize to your internal NTP
servers. You’ll save bandwidth and not abuse upstream resources.
3 to 5 upstream NTP sources provides optimum synchronization and protection
against failure scenarios. If you are using pool servers be sure to use a pool
who’s members are geographically close to your machines. The vendor pool your
Linux distro comes with is not geographically close to you. For example, I
When building your internal infrastructure your
machines should sync with 3 (perhaps more in some situations) of your own
internal NTP servers. Your internal NTP server should sync with 5 upstream
Depending on your resources one or more of which may be a GPS or atomic time
source that your place of business can easily acquire and install. Remember,
you need at least 3 sources and buying and installing 3 time sources in
different data centers (use different brands and sources – not all identical
GPS sources) does get expensive. A really good compromise is to buy one time
source and use 4 other time upstreams on different Internet networks.
Never, ever use only 2 upstream sources. Let’s look at the failure conditions.
Remember, the most common failure condition is that an NTP source is sending
the wrong time. Its easy to assume that the most common failure situation is
a non-responding upstream – that’s probably the second most common.
- One upstream is a single point of failure. You are guaranteeing that your
NTP infrastructure will just simply have the wrong time.
- Worst case. The NTP algorithm (or humans for that matter) cannot look at
two time sources that differ and reliably choose the correct one. Its a 50%
- Minimal reliable configuration.
- Can tolerate the loss of an NTP server and still have sufficient data
to detect falsetickers.
Your business’s NTP servers should use stratum 1 or 2 servers from different
networks. You should include your ISP’s NTP server if available to better
withstand network outages. If you have your own reference source then, of
course, you should have that in your NTP server’s configuration. If you
do have your own stratum 1 source you might set your NTP servers to perfer
that source. Do not let ordinary machines sync directly from your stratum
Like any service you need to monitor your NTP servers for health. Pool
members change, NTP servers become overwhelmed. You may need to peridically
evaluate your NTP servers upstream sources.
Your NTP clients and servers should have the drift file configured. This
records the average drift of your machine’s internal clock compared to
the upstream time sources. It is used if your machine cannot reach any
NTP servers. NTP also uses this when the daemon first starts. This does
help in the failure condition of no reachable NTP servers. Hopefully,
a short lived failure condition.
A common issue I see is that the specified directory is missing or the NTP
daemon does not have permission to write here. The NTP user should own
this directory. Your configuration management system of choice should
NTP in VMs, Laptops, and Other Time Stealing Tech
There is a lot of miss-information about how to keep your VMs synchronized,
a lot of confusion, and a lot of drifting VMs. Of course, VMs will never
be a quality time source – that’s not our goal. Our goal is to reduce
the amount that the VM’s internal clock is stepped. (Or completely reset
due to a large time difference.) We want our adjustments to slew the
clock – this makes a specific second take slightly longer or shorter.
Stepping the clock can adversely affect some applications. But in any case
we want our VMs to continuously move toward synchronization and not be outside
500 to 1000 milliseconds of sync.
I use NTP on my VMs. VMWare recommends it. Amazon EC2 (Xen HVM) recommends
it. Vendors that say they can sync your VM for you are fewer and fewer.
Actually, I use the same NTP configuration on all of my servers and keep them
identical in this case. With any time-stealing technology you need to instruct
your NTP daemon not to panic when it discovers large time differences. NTP has
a “safety feature” that is what causes so much pain with keeping VMs in sync
and I turn that off.
tinker panic 0
See What Happens If the Reference Time Changes. This is what VMs
commonly experience and what I turn off.
A Basic Configuration
So, a basic NTP configuration that I might use on a machine not part of a
larger infrastructure is below. This will work on VMs. This trusts
the time sources (which has its own article’s worth of ramifications).
# General options
tinker panic 0
## Make sure this directory is owned by NTP
# NTP Server Infrastructure
# Access restrictions for this machine
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
A Final Note
If you got here, you should really be reading the NTP documentation: