Communications of the ACM
Securing Network Time Protocol
Credit: Rene Jansa
In the late 1970s David L. Mills began working on the problem of synchronizing time on networked computers, and Network Time Protocol (NTP) version 1 made its debut in 1980. This was when the Net was a much friendlier place—the ARPANET days. NTP version 2 appeared approximately one year later, about the same time as Computer Science Network (CSNET). National Science Foundation Network (NSFNET) launched in 1986. NTP version 3 showed up in 1993.
Depending on where you draw the line, the Internet became useful in 1991–1992 and fully arrived in 1995. NTP version 4 appeared in 1997. Now, 18 years later, the Internet Engineering Task Force (IETF) is almost done finalizing the NTP version 4 standard, and some of us are starting to think about NTP version 5.
All of this is being done by volunteers—with no budget, just by the good graces of companies and individuals who care. This is not a sustainable situation. Network Time Foundation (NTF) is the vehicle that can address this problem, with the support of other organizations and individuals. For example, the Linux Foundation's Core Infrastructure Initiative recently started partially funding two NTP developers: Poul-Henning Kamp for 60% of his available time to work on NTP, and me for 30%–50% of my NTP development work. (Please visit http://nwtime.org/ to see who is supporting Network Time Foundation.)
On the public Internet, NTP tends to be visible from three types of machines. One is in embedded systems. When shipped misconfigured by the vendor, these systems have been the direct cause of abuse (http://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse). These systems do not generally support external monitoring, so they are not generally abusable in the context of this article. The second set of machines would be routers, and the majority of the ones that run NTP are from Cisco and Juniper. The third set of machines tend to be Windows machines that run win32time (which does not allow monitoring, and is therefore neither monitorable, nor abusable in this context), and Unix boxes that run NTP, acting as local time servers and distributing time to other machines on the LAN that run NTP to keep the local clock synchronized.
For the first 20 years of NTP's history, these local time servers were often old, spare machines that ran a dazzling array of operating systems. Some of these machines kept much better time than others, and people would eventually run them as their master time servers. This is one of the main reasons the NTP codebase stuck with K&R C (named for its authors Brian Kernighan and Dennis Ritchie) for so many years, as that was the only easily available compiler on some of these older machines.
It was not until December 2006 that NTP upgraded its codebase from K&R C to ANSI C. For a good while, only C89 was required. This was a full six years beyond Y2K, when a lot of these older operating systems were obsolete but still in production. By this time, however, the hardware on which NTP was "easy" to run had changed to x86 gear, and gcc (GNU Compiler Collection) was the easy compiler choice.
The NTP codebase does its job very well, is very reliable, and has had an enviable record as far as security problems go. Companies and people often run ancient versions of this software on some embedded systems that effectively never get upgraded and run well enough for a very long time.
People just expect accurate time, and they rarely see the consequences of inaccurate time. If the time is wrong, it is often more important to fix it fast and then—maybe—see if the original problem can be identified. The odds of identifying the problem increase if it happens with any frequency. Last year, NTP and our software had an estimated one trillion hours plus of operation. We have received some bug reports over this interval, and we have some open bug reports we would love to resolve, but in spite of this, NTP generally runs very, very well.
Having said all of this, I should reemphasize that NTP made its debut in a much friendlier environment, and that if there was a problem with the time on a machine, it was important to fix the problem as quickly as possible. Over the years, this translated into making it easy for people to query an NTP instance to see what it has been doing. There are two primary reasons for this: one is so it is easy to see if the remote server you want to sync with is configured and behaving adequately; the other is so it is easy to get help from others if there is a problem.
While we have been taking steps over the years to make NTP more secure and immune to abuse, the public Internet had more than seven million abusable NTP servers in the fall of last year. As a result of people upgrading software, fixing configuration files, or because, sadly, some ISPs and IXPs have decided to block NTP traffic, the number of abusable servers has dropped by almost 99% in just a few months. This is a remarkably large and fast decline, until you realize that around 85,000 abusable servers still exist, and a DDoS (distributed denial-of-service) attack in the range of 50Gbps–400Gbps can be launched using 5,000 servers. There is still a lot of cleanup to be done.
One of the best and easiest ways of reducing and even eliminating DDoS attacks is to ensure computers on your networks send packets that come from only your IP space. To this end, you should visit http://www.bcp38.info/ and take steps to implement this practice for your networks, if you have not already done so.
As I mentioned, NTP runs on the public Internet in three major places: Embedded devices; Unix and some Windows computers; and Cisco and Juniper routers. Before we take a look at how to configure the latter two groups so they cannot be abused, let's look at the NTP release history.
NTP Release History
David L. Mills, now a professor emeritus and adjunct professor at the University of Delaware, gave us NTP version 1 in 1980. It was good, and then it got better. A new "experimental" version, xntp2, installed the main binary as xntpd, because, well, that was the easy way to keep the previous version and new version on a box at the same time. Then version 2 became stable and a recommended standard (RFC 1119), so work began on xntp3. But the main program was still installed as xntpd, even though the program was not really "experimental." Note that RFC-1305 defines NTPv3, but that standard was never finalized as a recommended standard—it remained a draft/elective standard. The RFC for NTPv4 is still in development but is expected to be a recommended standard.
As for the software release numbering, three of the releases from Mills are xntp3.3wx, xntp3.3wy, and xntp3.5f. These date from just after the time I started using NTP heavily, and I was also sending in portability patches. Back then, you unpacked the tarball, manually edited a config. local file, and did interesting things with the makefile to get the code to build. While Perl's metaconfig was available then and was great for poking around a system, it did not support subdirectory builds and thus could not use a single set of source code for multiple targets.
GNU autoconf was still pretty new at that time, and while it did not do nearly as good a job at poking around, it did support subdirectory builds. xntp3.5f was released just as I volunteered to convert the NTP code base to GNU autoconf. As part of that conversion, Mills and I discussed the version numbers, and he was OK with my releasing the first cut of the GNU autoconf code as xntp3-5.80. These were considered alpha releases, as .90 and above were reserved for beta releases. The first production release for this code would be xntp3-6.0, the sixth major release of NTPv3, except that shortly after xntp3-5.93e was released in late November 1993, Mills decided the NTPv3 code was good enough and that it was time to start on NTPv4.
At that point, I noticed many people had problems with the version-numbering scheme, as the use of both the dash (-) and dot (.) characters really confused people. So ntp-4.0.94a was the first beta release of the NTPv4 code in July 1997. The release numbers went from ntpPROTO-Maj.Min to ntp-PROTO. Maj.Min.
While this change had the desired effect of removing confusion about how to type the version number, it meant most people did not realize going from ntp-4.1.x to 4.2.x was a major release upgrade. People also did not seem to understand just how many items were being fixed or improved in minor releases. For more information about this, see the accompanying table.
At one point I tried going back to a version-numbering scheme that was closer to the previous method, but I got a lot of pushback so I did not go through with it. In hindsight, I should have stood my ground. Having seen how people do not appreciate the significance of the releases—major or minor—we will go back to a numbering scheme much closer to the original after 4.2.8 is released. The major release after ntp-4.2.8 will be something like ntp4-5.0.0 (or ntp-PROTO-Maj.Min.Point, if we keep the Major Minor numbers) or ntp4-3.0 (or ntpPROTO-Release.Point, if we go to a single release number from the current Major and Minor release numbers). Our source archives reveal how the release numbering choices have evolved over the years, and how badly some of them collated.
Securing NTP
Before we delve into how to secure NTP, I recommend you listen to Dan Geer's keynote speech for Blackhat 2014, if you have not already done so (https://www.youtube.com/watch?v=nT-TGvYOBpI). It will be an excellent use of an hour of your time. If you watch it and disagree with what he says, then I wonder why you are reading this article to look for a solution to NTP abuse vector problems.
Now, to secure NTP, first implement BCP38 (http://www.bcp38.info). It is not that difficult.
If you want to ensure NTP on your Cisco or Juniper routers is protected, then consult their documentation on how to do so. You will find lots of good discussions and articles on the Web with additional updated information, and I recommend http://www.team-cymru.org/ReadingRoom/Templates/secure-ntp-template.html for information about securing NTP on Cisco and Juniper routers.
The NTP support site provides information on how to secure NTP through the ntp.conf file. Find some discussion and a link to that site at http://nwtime.org/ntp-winter-2013-network-drdos-attacks/. NTF is also garnering the resources to produce an online ntp.conf generator that will implement BCP for this file and make it easy to update that file as our codebase and knowledge evolves.
That said, the most significant NTP abuse vectors are disabled by default starting with ntp-4.2.7p27, and these and other improvements will be in ntp-4.2.8, which was released at press time.
For versions 4.2.6 through 4.2.7p27, this abuse vector can be prevented by adding the following to your ntp.conf file:
Note that if you have additional restrict lines for IPs or networks that do not include noquery restriction, ask yourself if it is possible for those IPs to be spoofed.
For version 4.2.4, which was released in December 2006 and EOLed (brought to the end-of-life) in December 2009, consider the following:
- You did not pay attention to what Dan Geer said.
- Did you notice we fixed 630-1,000 issues going from 4.2.4 to 4.2.6?
- Are you still interested in running 4.2.4? Do you really have a good reason for this?
If so, add to your ntp.conf file:
For version 4.2.2, which was released in June 2006 and EOLed in December 2006:
- You did not pay attention to what Dan Geer said.
- Did you notice we fixed about 450 issues going from 4.2.2 to 4.2.4, and 630–1,000 issues going from 4.2.4 to 4.2.6? That is between 1,000 and 1,500 issues. Seriously.
- Are you still interested in running 4.2.2? Do you really have a good reason for this?
If so, add to your ntp.conf file:
For version 4.2.0, which was released in 2003 and EOLed in 2006:
- You did not pay attention to what Dan Geer said.
- Did you notice we fixed about 560 issues going from 4.2.0 to 4.2.2, 450 issues going from 4.2.2 to 4.2.4, and 630–1,000 issues going from 4.2.4 to 4.2.6? That is between 1,500 and 2,000 issues. Seriously.
- Are you still interested in running 4.2.2? Do you really have a good reason for this?
If so, add to your ntp.conf file:
For versions 4.0 through 4.1.1, which were released and EOLed somewhere around 2001 to 2003, no numbers exist for how many issues were fixed in these releases:
- You did not pay attention to what Dan Geer said.
- There are probably in excess of 2,000–2,500 issues fixed since then.
- Are you still interested in running 4.0 or 4.1 code? Do you really have a good reason for this?
If so, add to your ntp.conf file:
Now let's talk about xntp3, which was EOLed in September 1997. Do the math on how old that is, take a guess at how many issues have been fixed since then, and ask yourself and anybody else who has a voice in the matter: Why are you running software that was EOLed 17 years ago, when thousands of issues have been fixed and an improved protocol has been implemented since then?
If your answer is: "Because NTPv3 was a standard and NTPv4 is not yet a standard," then I have bad news for you. NTPv3 was not a recommended standard; it was only a draft/elective standard. If you really want to run only officially standard software, you can drop back to NTPv2—and I do not know anybody who would want to do that.
If your answer is: "We're not sure how stable NTPv4 is," then I will point out that NTPv4 has an estimated 5–10 trillion operational hours at this point. How much more do you want?
But if you insist, the way to secure xntp2 and xntp3 against the described abuse vector is to add to your ntp.conf file:
Related articles
on queue.acm.org
Principles of Robust Timing over the Internet
Julien Ridoux and Darryl Veitch
http://queue.acm.org/detail.cfm?id=1773943
Toward Higher Precision
Rick Ratzel and Rodney Greenstreet
http://queue.acm.org/detail.cfm?id=2354406
The One-second War (What Time Will You Die?)
Poul-Henning Kamp
http://queue.acm.org/detail.cfm?id=1967009
The Digital Library is published by the Association for Computing Machinery. Copyright © 2015 ACM, Inc.