Managing NTP at Scale on VMs
One of my favorite services to run is NTP. The math that makes it work is elegant, graphing NTP’s performance produces beautiful graphs, and NTP is usually a low maintenance service. Most importantly, accurate time synchronization is crucial to every day IT functions. This should be something in every Operation Engineer’s tool bag.
Yet, its always difficult – socially – to make changes to an NTP infrastructure. Every client I’ve worked with has been hesitant to allow changes to their NTP configuration. Many assume that NTP is “simple” and “working” why should it be changed?
Don’t assume that NTP is “simple” and ignore it. Like everything else, one needs a good understanding of how it works to have accurate time synchronization. Here are some tips for running an NTP infrastructure that maintains accurate synchronization and works on all of your machines – including those often drifting virtual machines. At the bottom you’ll find an example NTP configuration.
UTC Is Your Friend
The first tip for setting up your infrastructure to have reasonably accurate time is to set your BIOS clock to UTC. UTC doesn’t have Daylight Savings of other weird time changes. Do not keep your BIOS clock in the local time.
Hierarchy of NTP
NTP servers build a hierarchy and the level you are on in that hierarchy is called your stratum. The smaller the stratum, the more accurate your synchronization should be. However, there are many thousands (or more) of these hierarchies. You can use more than one (and you should). They may overlap upstream from your servers. This is used to create a large amount of redundancy. But, it can also cause hidden single points of failure. Do some work to identify your upstream sources.
For one or a handful of servers, VMs, or workstations directly connected to the
Internet, using NTP sources from a pool is best. Your distro most likely
has a good starting point for your NTP configuration in /etc/ntp.conf.
For more machines or machines on a backend network you want to setup your own bit of NTP hierarchy. Normal machines will synchronize to your internal NTP servers. You’ll save bandwidth and not abuse upstream resources.
Upstream Sources
3 to 5 upstream NTP sources provides optimum synchronization and protection against failure scenarios. If you are using pool servers be sure to use a pool who’s members are geographically close to your machines. The vendor pool your Linux distro comes with is not geographically close to you. For example, I might use:
server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org
When building your internal infrastructure your machines should sync with 3 (perhaps more in some situations) of your own internal NTP servers. Your internal NTP server should sync with 5 upstream sources.
Depending on your resources one or more of which may be a GPS or atomic time source that your place of business can easily acquire and install. Remember, you need at least 3 sources and buying and installing 3 time sources in different data centers (use different brands and sources – not all identical GPS sources) does get expensive. A really good compromise is to buy one time source and use 4 other time upstreams on different Internet networks.
Never, ever use only 2 upstream sources. Let’s look at the failure conditions. Remember, the most common failure condition is that an NTP source is sending the wrong time. Its easy to assume that the most common failure situation is a non-responding upstream – that’s probably the second most common.
- One upstream is a single point of failure. You are guaranteeing that your NTP infrastructure will just simply have the wrong time.
- Worst case. The NTP algorithm (or humans for that matter) cannot look at two time sources that differ and reliably choose the correct one. Its a 50% guess.
- Minimal reliable configuration.
- Can tolerate the loss of an NTP server and still have sufficient data to detect falsetickers.
Your business’s NTP servers should use stratum 1 or 2 servers from different networks. You should include your ISP’s NTP server if available to better withstand network outages. If you have your own reference source then, of course, you should have that in your NTP server’s configuration. If you do have your own stratum 1 source you might set your NTP servers to perfer that source. Do not let ordinary machines sync directly from your stratum 1 source.
Like any service you need to monitor your NTP servers for health. Pool members change, NTP servers become overwhelmed. You may need to peridically evaluate your NTP servers upstream sources.
Drift Files
Your NTP clients and servers should have the drift file configured. This records the average drift of your machine’s internal clock compared to the upstream time sources. It is used if your machine cannot reach any NTP servers. NTP also uses this when the daemon first starts. This does help in the failure condition of no reachable NTP servers. Hopefully, a short lived failure condition.
driftfile /var/lib/ntp/ntp.drift
A common issue I see is that the specified directory is missing or the NTP daemon does not have permission to write here. The NTP user should own this directory. Your configuration management system of choice should enforce this.
NTP in VMs, Laptops, and Other Time Stealing Tech
There is a lot of miss-information about how to keep your VMs synchronized, a lot of confusion, and a lot of drifting VMs. Of course, VMs will never be a quality time source – that’s not our goal. Our goal is to reduce the amount that the VM’s internal clock is stepped. (Or completely reset due to a large time difference.) We want our adjustments to slew the clock – this makes a specific second take slightly longer or shorter. Stepping the clock can adversely affect some applications. But in any case we want our VMs to continuously move toward synchronization and not be outside 500 to 1000 milliseconds of sync.
I use NTP on my VMs. VMWare recommends it. Amazon EC2 (Xen HVM) recommends it. Vendors that say they can sync your VM for you are fewer and fewer. Actually, I use the same NTP configuration on all of my servers and keep them identical in this case. With any time-stealing technology you need to instruct your NTP daemon not to panic when it discovers large time differences. NTP has a “safety feature” that is what causes so much pain with keeping VMs in sync and I turn that off.
tinker panic 0
See What Happens If the Reference Time Changes. This is what VMs commonly experience and what I turn off.
A Basic Configuration
So, a basic NTP configuration that I might use on a machine not part of a larger infrastructure is below. This will work on VMs. This trusts the time sources (which has its own article’s worth of ramifications).
# General options
tinker panic 0
## Make sure this directory is owned by NTP
driftfile /var/lib/ntp/drift
# NTP Server Infrastructure
server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org
# Access restrictions for this machine
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
restrict 127.0.0.1
restrict ::1
A Final Note
If you got here, you should really be reading the NTP documentation: